2024-03-25T16:10:13.273 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:10:13.274 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/orig.config.yaml 2024-03-25T16:10:13.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:10:13.406 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621253' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGgMmjL9vIjb25dSI+q3qzntVgzJ5w3kec8X6xKB/SOOyIFMDrMxKtF8cO1JbTgGDF3xdWS63ZPWI7+5Td7VeE= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsZiptMAL9EmMGCc4pX0U3Pjy3haBJUASvF1lidQ3R5LIIt+aMAsppp9t5Nrftpp9rpOejdZcHctBsAY5nXXc4= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:10:13.407 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:10:13.407 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:10:13.407 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:10:13.409 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:10:13.434 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:10:13.434 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:10:13.434 INFO:teuthology.packaging:ref: None 2024-03-25T16:10:13.434 INFO:teuthology.packaging:tag: None 2024-03-25T16:10:13.434 INFO:teuthology.packaging:branch: main 2024-03-25T16:10:13.434 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:13.434 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:10:13.657 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:10:13.659 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:10:13.667 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:10:13.667 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:10:13.681 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:10:13.702 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:10:13.709 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:10:13.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:05.077750', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGgMmjL9vIjb25dSI+q3qzntVgzJ5w3kec8X6xKB/SOOyIFMDrMxKtF8cO1JbTgGDF3xdWS63ZPWI7+5Td7VeE='} 2024-03-25T16:10:13.754 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:05.076152', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsZiptMAL9EmMGCc4pX0U3Pjy3haBJUASvF1lidQ3R5LIIt+aMAsppp9t5Nrftpp9rpOejdZcHctBsAY5nXXc4='} 2024-03-25T16:10:13.754 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:10:13.761 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:10:13.761 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:10:13.761 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:10:13.830 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f581ff8d0d0>, signals=[15]) 2024-03-25T16:10:13.831 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:10:13.836 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:10:13.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T16:10:13.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:13.943 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T16:10:13.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:14.032 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:10:14.043 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-03-25T16:10:14.060 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-03-25T16:10:14.061 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-03-25T16:10:14.118 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-03-25T16:10:14.118 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-03-25T16:10:14.118 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:14.119 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:14.120 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-03-25T16:10:14.149 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-03-25T16:10:14.167 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-03-25T16:10:14.167 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:14.225 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:14.226 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:14.226 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-03-25T16:10:14.255 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:10:14.265 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:10:14.271 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:10:14.271 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:14.273 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:14.289 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:10:14.295 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:10:14.295 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:14.330 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:14.417 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:10:14.470 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:10:14.484 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:10:14.484 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:10:14.485 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:14.485 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:14.502 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-25T16:10:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:14.503 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-25T16:10:14.560 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:14.560 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-361.el9.x86_64 2024-03-25T16:10:14.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-03-25T16:10:14.563 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:14.563 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-361.el9.x86_64 2024-03-25T16:10:14.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-03-25T16:10:16.678 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:10:16.703 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:10:18.370 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T16:10:18.479 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-25T16:10:19.402 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:10:19.486 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:10:23.706 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-03-25T16:10:27.527 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:07 2024-03-25T16:10:27.554 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-25T16:10:29.088 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-25T16:10:31.268 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T16:10:31.836 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-25T16:10:33.245 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-25T16:10:34.947 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T16:10:36.256 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:10:38.404 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:10:38.405 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:10:38.493 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:10:38.497 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-03-25T16:10:38.498 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:10:38.499 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 76 M 2024-03-25T16:10:38.499 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 123 M 2024-03-25T16:10:38.499 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:10:38.632 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 621 kB/s | 44 kB 00:00 2024-03-25T16:10:39.224 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-25T16:10:39.808 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-25T16:10:40.022 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:10:40.050 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-25T16:10:40.050 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:10:40.050 INFO:teuthology.orchestra.run.smithi164.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-25T16:10:40.718 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:10:41.082 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:10:41.082 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:10:42.115 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:10:42.164 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:10:42.262 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:10:42.262 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:10:42.362 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:10:42.366 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:10:42.367 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:10:42.367 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:10:42.367 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:10:42.367 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:10:42.367 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:10:42.368 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:10:42.368 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-03-25T16:10:42.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:10:42.369 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 76 M 2024-03-25T16:10:42.369 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 123 M 2024-03-25T16:10:42.369 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:10:42.773 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 339 kB/s | 44 kB 00:00 2024-03-25T16:10:44.585 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:10:45.253 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:10:45.341 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:10:45.790 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.7 MB/s | 15 MB 00:03 2024-03-25T16:10:47.499 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.8 MB/s | 27 MB 00:04 2024-03-25T16:10:48.103 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:10:48.244 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:10:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 4.8 MB/s | 33 MB 00:06 2024-03-25T16:10:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:10:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-25T16:10:50.324 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:10:50.748 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:10:50.748 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:10:51.805 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:10:51.805 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:10:52.940 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:10:52.977 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:54.410 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:10:55.045 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:10:55.139 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:10:57.732 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:58.024 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:10:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:03.237 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:03.282 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:08.371 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:26.986 INFO:teuthology.orchestra.run.smithi164.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:11:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:11:27.030 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:27.426 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:27.426 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:27.426 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:27.426 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:27.854 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:11:27.855 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:11:28.189 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:28.285 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:28.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:28.285 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:11:28.285 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:11:28.285 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:11:28.286 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:11:28.286 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:11:28.286 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:28.365 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:28.366 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:28.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:11:28.402 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:11:28.403 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:11:28.404 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-03-25T16:11:28.404 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:11:28.404 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:11:28.405 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:11:28.405 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:11:28.405 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:11:29.056 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-03-25T16:11:30.437 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:11:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-03-25T16:11:30.476 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-25T16:11:30.498 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.Kf7Xq2k5ED 2024-03-25T16:11:30.498 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Kf7Xq2k5ED 2024-03-25T16:11:30.567 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.Kf7Xq2k5ED 2024-03-25T16:11:30.701 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.Kf7Xq2k5ED is 6KB 2024-03-25T16:11:30.751 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.Kf7Xq2k5ED 2024-03-25T16:11:30.765 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:11:30.839 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:11:30.839 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:11:30.839 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:11:30.839 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:11:30.840 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:11:30.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-03-25T16:11:39.356 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:39.356 INFO:teuthology.orchestra.run.smithi046.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:11:39.357 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:11:39.402 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:39.767 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:39.768 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.203 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.204 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:11:40.204 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:11:40.530 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:40.633 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.633 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:40.633 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:11:40.633 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:11:40.633 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:11:40.634 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:11:40.634 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:11:40.634 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:40.717 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:40.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:40.717 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-03-25T16:11:40.765 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-03-25T16:11:40.765 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-03-25T16:11:40.765 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-03-25T16:11:40.765 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-03-25T16:11:40.765 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:11:40.766 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:11:40.767 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:11:40.767 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:11:40.767 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-03-25T16:11:40.767 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:11:40.767 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:11:40.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:11:40.768 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:11:40.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:11:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-03-25T16:11:42.940 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18091: grub2-probe 2024-03-25T16:11:42.940 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18091: grub2-probe 2024-03-25T16:11:42.966 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18091: grub2-probe 2024-03-25T16:11:42.966 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18091: grub2-probe 2024-03-25T16:11:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18139: grub2-probe 2024-03-25T16:11:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18139: grub2-probe 2024-03-25T16:11:43.231 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18139: grub2-probe 2024-03-25T16:11:43.231 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18139: grub2-probe 2024-03-25T16:11:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18187: grub2-probe 2024-03-25T16:11:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18187: grub2-probe 2024-03-25T16:11:43.401 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18187: grub2-probe 2024-03-25T16:11:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18187: grub2-probe 2024-03-25T16:11:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18235: grub2-probe 2024-03-25T16:11:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18235: grub2-probe 2024-03-25T16:11:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18235: grub2-probe 2024-03-25T16:11:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18235: grub2-probe 2024-03-25T16:11:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18283: grub2-probe 2024-03-25T16:11:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18283: grub2-probe 2024-03-25T16:11:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28580]) leaked on vgs invocation. Parent PID 18283: grub2-probe 2024-03-25T16:11:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[29305]) leaked on vgs invocation. Parent PID 18283: grub2-probe 2024-03-25T16:11:43.919 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:11:43.933 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-03-25T16:11:43.935 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-03-25T16:11:43.950 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.TkeCMY3BZd 2024-03-25T16:11:43.950 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TkeCMY3BZd 2024-03-25T16:11:44.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.TkeCMY3BZd 2024-03-25T16:11:44.158 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.TkeCMY3BZd is 6KB 2024-03-25T16:11:44.206 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.TkeCMY3BZd 2024-03-25T16:11:44.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:11:44.290 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:11:44.290 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:11:44.290 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:11:44.290 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:11:44.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:11:44.424 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-03-25T16:12:01.010 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:12:01.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:12:01.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:14.438 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:12:14.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:12:14.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:19.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-25T16:12:28.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:12:28.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:31.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-25T16:12:32.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-25T16:12:41.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:12:41.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:43.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:12:43.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:44.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-25T16:12:57.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:12:57.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:01.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-25T16:13:15.423 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-25T16:13:16.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:13:16.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:18.619 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-25T16:13:19.112 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:13:19.112 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:13:19.112 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-25T16:13:19.126 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:13:19.126 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:13:19.126 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:13:19.126 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:13:20.128 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:13:20.128 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:13:20.228 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-03-25T16:13:20.263 DEBUG:teuthology.parallel:result is None 2024-03-25T16:13:30.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:13:30.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:30.778 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-25T16:13:31.212 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:13:31.212 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:13:31.212 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-25T16:13:31.226 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:13:31.226 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:13:31.226 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:13:31.226 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:13:32.228 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:13:32.228 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:13:32.317 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-03-25T16:13:32.358 DEBUG:teuthology.parallel:result is None 2024-03-25T16:13:32.358 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:13:32.366 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:13:32.367 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:13:32.369 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:13:32.387 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:13:32.394 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:13:32.404 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:13:32.405 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:13:32.424 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:13:32.514 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:13:32.522 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:13:32.523 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:13:32.525 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:13:32.558 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:13:32.566 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:13:32.572 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:13:32.579 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:13:32.581 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:13:32.593 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:13:32.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:13:32.624 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:13:32.653 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:13:32.664 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:13:32.664 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:13:32.692 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:13:32.709 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:13:32.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:13:32.829 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:13:32.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:13:32.954 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T16:13:32.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:13:33.023 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:13:33.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:13:33.122 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:13:33.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:13:33.249 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T16:13:33.249 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:13:33.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-03-25T16:13:33.319 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-03-25T16:13:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:13:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:13:33.838 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:13:33.845 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:13:33.845 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:13:33.857 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:13:33.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-03-25T16:13:33.927 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-03-25T16:13:33.929 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-03-25T16:13:34.021 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-03-25T16:13:34.024 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:13:34.024 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-03-25T16:13:34.048 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-03-25T16:13:34.048 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-03-25T16:13:34.073 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-03-25T16:13:34.073 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:13:34.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:13:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:13:34.117 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:13:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:13:34.148 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:13:34.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:13:34.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:13:34.251 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:13:34.267 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:13:34.291 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:13:34.294 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:17:56.931 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-03-25T16:17:56.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:17:56.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:57.025 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-25T16:17:57.106 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-25T16:17:57.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:17:57.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:57.196 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-25T16:17:57.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-25T16:17:57.236 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:17:57.248 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:17:57.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:17:57.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:17:57.251 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:17:57.251 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:17:57.281 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:17:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:17:57.306 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:17:57.326 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:17:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-03-25T16:17:57.341 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-03-25T16:17:57.353 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-03-25T16:17:57.357 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:17:57.366 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-03-25T16:17:57.373 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:17:57.383 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:17:57.399 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.428 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:17:57.452 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:17:57.452 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-25T16:17:57.452 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.453 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.453 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.453 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:17:57.454 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:17:57.462 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:17:57.462 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:17:57.462 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:17:57.462 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:17:57.468 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:17:57.468 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:17:57.468 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:17:57.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:17:57.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:17:57.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:17:57.620 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:17:57.621 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:17:57.693 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:17:57.693 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:17:58.014 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:17:58.014 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T16:17:58.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:17:58.044 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:17:58.045 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:17:58.087 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:17:58.087 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T16:17:58.088 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:17:58.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:17:58.118 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:17:58.118 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:17:58.188 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:17:58.197 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:17:58.265 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-03-25T16:17:58.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-03-25T16:17:58.271 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:17:58.338 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-03-25T16:17:58.339 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-03-25T16:17:58.528 INFO:teuthology.orchestra.run.smithi046.stdout:74 files removed 2024-03-25T16:17:58.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw 2024-03-25T16:17:58.588 INFO:teuthology.orchestra.run.smithi164.stdout:74 files removed 2024-03-25T16:17:58.615 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw 2024-03-25T16:17:59.749 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 92 kB/s | 82 kB 00:00 2024-03-25T16:18:00.608 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:18:00.657 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 47 kB/s | 82 kB 00:01 2024-03-25T16:18:01.408 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:18:01.500 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:18:02.183 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T16:18:02.403 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.7 MB 00:00 2024-03-25T16:18:03.080 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.7 MB 00:00 2024-03-25T16:18:03.869 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-25T16:18:04.628 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T16:18:05.468 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-25T16:18:07.161 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-25T16:18:08.193 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-25T16:18:11.270 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-25T16:18:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-25T16:18:13.750 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:18:18.672 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-25T16:18:21.125 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:18:21.140 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:21.140 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:18:21.140 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:21.141 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:21.142 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-03-25T16:18:21.143 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:Install 30 Packages 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 58 M 2024-03-25T16:18:21.144 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:18:23.481 INFO:teuthology.orchestra.run.smithi164.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-25T16:18:23.590 INFO:teuthology.orchestra.run.smithi164.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-25T16:18:23.981 INFO:teuthology.orchestra.run.smithi164.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 742 kB 00:00 2024-03-25T16:18:24.274 INFO:teuthology.orchestra.run.smithi164.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.7 MB/s | 498 kB 00:00 2024-03-25T16:18:25.957 INFO:teuthology.orchestra.run.smithi164.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.1 MB/s | 9.7 MB 00:04 2024-03-25T16:18:26.057 INFO:teuthology.orchestra.run.smithi164.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:18:26.158 INFO:teuthology.orchestra.run.smithi164.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-25T16:18:26.258 INFO:teuthology.orchestra.run.smithi164.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-25T16:18:26.441 INFO:teuthology.orchestra.run.smithi164.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.7 MB/s | 320 kB 00:00 2024-03-25T16:18:26.617 INFO:teuthology.orchestra.run.smithi164.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-25T16:18:26.717 INFO:teuthology.orchestra.run.smithi164.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 995 kB/s | 99 kB 00:00 2024-03-25T16:18:26.909 INFO:teuthology.orchestra.run.smithi164.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 552 kB/s | 106 kB 00:00 2024-03-25T16:18:27.001 INFO:teuthology.orchestra.run.smithi164.stdout:(13/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 5.1 MB 00:02 2024-03-25T16:18:27.026 INFO:teuthology.orchestra.run.smithi164.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T16:18:27.068 INFO:teuthology.orchestra.run.smithi164.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 659 kB/s | 44 kB 00:00 2024-03-25T16:18:27.302 INFO:teuthology.orchestra.run.smithi164.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 661 kB 00:00 2024-03-25T16:18:27.327 INFO:teuthology.orchestra.run.smithi164.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-03-25T16:18:27.352 INFO:teuthology.orchestra.run.smithi164.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-25T16:18:27.969 INFO:teuthology.orchestra.run.smithi164.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 56 kB/s | 35 kB 00:00 2024-03-25T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 232 kB/s | 216 kB 00:00 2024-03-25T16:18:28.512 INFO:teuthology.orchestra.run.smithi164.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T16:18:28.678 INFO:teuthology.orchestra.run.smithi164.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 63 kB/s | 44 kB 00:00 2024-03-25T16:18:28.895 INFO:teuthology.orchestra.run.smithi164.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 120 kB/s | 26 kB 00:00 2024-03-25T16:18:28.987 INFO:teuthology.orchestra.run.smithi164.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-03-25T16:18:29.021 INFO:teuthology.orchestra.run.smithi164.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-25T16:18:29.056 INFO:teuthology.orchestra.run.smithi164.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 982 kB/s | 67 kB 00:00 2024-03-25T16:18:29.122 INFO:teuthology.orchestra.run.smithi164.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.9 MB/s | 192 kB 00:00 2024-03-25T16:18:29.213 INFO:teuthology.orchestra.run.smithi164.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-25T16:18:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-25T16:18:30.456 INFO:teuthology.orchestra.run.smithi164.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 3.1 MB 00:01 2024-03-25T16:18:30.717 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 900 kB/s | 19 MB 00:21 2024-03-25T16:18:31.806 INFO:teuthology.orchestra.run.smithi164.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-03-25T16:18:32.589 INFO:teuthology.orchestra.run.smithi164.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 20 MB 00:11 2024-03-25T16:18:32.592 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:32.593 INFO:teuthology.orchestra.run.smithi164.stdout:Total 5.1 MB/s | 58 MB 00:11 2024-03-25T16:18:32.628 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:18:32.667 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:18:32.667 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:18:33.151 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:18:33.151 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:18:34.610 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:18:34.711 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:18:34.808 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:18:34.861 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:18:34.884 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:18:34.901 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:18:35.076 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.1 MB 00:00 2024-03-25T16:18:35.094 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:18:35.148 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:18:35.483 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:18:35.546 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:18:35.736 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:18:35.764 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:18:35.827 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:18:35.856 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:18:35.899 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:18:35.968 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:18:35.991 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:18:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:18:36.081 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:18:36.193 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:18:36.241 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:18:36.290 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:18:36.322 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:18:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:18:36.384 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:18:36.469 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-03-25T16:18:36.700 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:18:36.730 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:18:36.803 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:18:36.835 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:18:36.879 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:18:37.334 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:18:37.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:18:37.420 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:18:37.488 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:18:37.527 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:18:37.537 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:18:39.719 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:18:39.757 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:18:40.317 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:18:40.360 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:18:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:18:40.455 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:18:40.455 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:18:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:18:42.696 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-25T16:18:45.211 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:45.223 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:45.224 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:45.225 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:18:45.226 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:18:45.227 INFO:teuthology.orchestra.run.smithi046.stdout:Install 30 Packages 2024-03-25T16:18:45.227 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2024-03-25T16:18:45.227 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:18:45.227 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 58 M 2024-03-25T16:18:45.227 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:18:49.786 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-03-25T16:18:49.787 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:49.955 INFO:teuthology.orchestra.run.smithi046.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 5.3 MB 00:04 2024-03-25T16:18:50.055 INFO:teuthology.orchestra.run.smithi046.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:18:50.698 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:18:50.722 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:18:50.722 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:50.722 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:18:50.723 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:18:50.723 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:18:50.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:50.724 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:18:50.746 INFO:teuthology.orchestra.run.smithi046.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 742 kB 00:00 2024-03-25T16:18:50.808 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:18:50.809 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:18:51.230 INFO:teuthology.orchestra.run.smithi046.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.0 MB/s | 498 kB 00:00 2024-03-25T16:18:51.764 INFO:teuthology.orchestra.run.smithi046.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.6 MB/s | 9.7 MB 00:06 2024-03-25T16:18:51.864 INFO:teuthology.orchestra.run.smithi046.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 453 kB/s | 45 kB 00:00 2024-03-25T16:18:51.927 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:18:51.927 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:18:51.927 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:18:51.927 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:18:51.927 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:18:51.928 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:18:51.929 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:18:51.930 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:18:51.930 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:18:51.930 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:18:51.930 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:18:51.947 INFO:teuthology.orchestra.run.smithi046.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-25T16:18:52.115 INFO:teuthology.orchestra.run.smithi046.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 972 kB/s | 162 kB 00:00 2024-03-25T16:18:52.431 INFO:teuthology.orchestra.run.smithi046.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 320 kB 00:00 2024-03-25T16:18:52.715 INFO:teuthology.orchestra.run.smithi046.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 298 kB 00:00 2024-03-25T16:18:52.882 INFO:teuthology.orchestra.run.smithi046.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 99 kB 00:00 2024-03-25T16:18:52.990 INFO:teuthology.orchestra.run.smithi046.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:18:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:53.043 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:18:53.044 INFO:teuthology.orchestra.run.smithi046.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:18:53.074 INFO:teuthology.orchestra.run.smithi046.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:18:53.141 INFO:teuthology.orchestra.run.smithi046.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:18:53.183 INFO:teuthology.orchestra.run.smithi046.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:18:53.217 INFO:teuthology.orchestra.run.smithi046.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:18:53.367 INFO:teuthology.orchestra.run.smithi046.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-25T16:18:53.388 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test 2024-03-25T16:18:53.417 INFO:teuthology.orchestra.run.smithi046.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-03-25T16:18:53.468 INFO:teuthology.orchestra.run.smithi046.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 893 kB/s | 44 kB 00:00 2024-03-25T16:18:53.947 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:35 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:18:54.095 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:18:54.095 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:54.095 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:18:54.095 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2024-03-25T16:18:54.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:18:54.097 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 48 M 2024-03-25T16:18:54.097 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 204 M 2024-03-25T16:18:54.097 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:18:54.270 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:18:54.379 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.7 MB/s | 300 kB 00:00 2024-03-25T16:18:54.404 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T16:18:54.596 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 981 kB/s | 187 kB 00:00 2024-03-25T16:18:54.668 INFO:teuthology.orchestra.run.smithi046.stdout:(21/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 5.1 MB 00:03 2024-03-25T16:18:54.930 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 206 kB/s | 163 kB 00:00 2024-03-25T16:18:58.727 INFO:teuthology.orchestra.run.smithi046.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 20 MB 00:13 2024-03-25T16:19:14.600 INFO:teuthology.orchestra.run.smithi046.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 1.6 kB/s | 26 kB 00:15 2024-03-25T16:19:14.717 INFO:teuthology.orchestra.run.smithi046.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-03-25T16:19:14.801 INFO:teuthology.orchestra.run.smithi046.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-25T16:19:15.018 INFO:teuthology.orchestra.run.smithi046.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-25T16:19:15.110 INFO:teuthology.orchestra.run.smithi046.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-25T16:19:15.285 INFO:teuthology.orchestra.run.smithi046.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-25T16:19:22.326 INFO:teuthology.orchestra.run.smithi046.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 491 kB/s | 3.4 MB 00:07 2024-03-25T16:19:27.584 INFO:teuthology.orchestra.run.smithi046.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 609 kB/s | 3.1 MB 00:05 2024-03-25T16:19:33.501 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 47 MB 00:39 2024-03-25T16:19:33.503 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:33.503 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.2 MB/s | 48 MB 00:39 2024-03-25T16:19:33.515 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:19:33.535 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:19:33.535 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:19:33.915 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:19:33.915 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:19:34.510 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:19:34.593 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:19:34.643 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:19:34.683 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:19:34.751 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:19:34.778 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:19:39.409 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:19:39.489 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:19:41.960 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:19:41.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:19:41.961 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:19:41.961 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:19:41.961 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:19:41.961 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:19:42.325 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.326 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:19:42.326 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:19:42.326 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:19:42.326 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:19:42.326 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:19:42.615 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph 2024-03-25T16:19:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:19:43.347 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:19:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:19:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:19:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout:Install 36 Packages 2024-03-25T16:19:43.353 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:19:43.354 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 30 M 2024-03-25T16:19:43.354 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 107 M 2024-03-25T16:19:43.354 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:19:44.272 INFO:teuthology.orchestra.run.smithi164.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T16:19:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 342 kB/s | 1.5 MB 00:04 2024-03-25T16:19:49.214 INFO:teuthology.orchestra.run.smithi164.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 411 kB/s | 2.1 MB 00:05 2024-03-25T16:19:50.172 INFO:teuthology.orchestra.run.smithi164.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 258 kB/s | 247 kB 00:00 2024-03-25T16:19:50.305 INFO:teuthology.orchestra.run.smithi164.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:19:50.347 INFO:teuthology.orchestra.run.smithi164.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:19:50.381 INFO:teuthology.orchestra.run.smithi164.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 943 kB/s | 32 kB 00:00 2024-03-25T16:19:50.415 INFO:teuthology.orchestra.run.smithi164.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:19:50.449 INFO:teuthology.orchestra.run.smithi164.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:19:50.491 INFO:teuthology.orchestra.run.smithi164.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:19:50.557 INFO:teuthology.orchestra.run.smithi164.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:19:50.591 INFO:teuthology.orchestra.run.smithi164.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:19:50.625 INFO:teuthology.orchestra.run.smithi164.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:19:50.667 INFO:teuthology.orchestra.run.smithi164.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:19:50.867 INFO:teuthology.orchestra.run.smithi164.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 624 kB/s | 125 kB 00:00 2024-03-25T16:19:50.942 INFO:teuthology.orchestra.run.smithi164.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-25T16:19:51.193 INFO:teuthology.orchestra.run.smithi164.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 604 kB/s | 151 kB 00:00 2024-03-25T16:19:51.268 INFO:teuthology.orchestra.run.smithi164.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-03-25T16:19:51.368 INFO:teuthology.orchestra.run.smithi164.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:19:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T16:19:51.477 INFO:teuthology.orchestra.run.smithi164.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-25T16:19:51.528 INFO:teuthology.orchestra.run.smithi164.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T16:19:51.578 INFO:teuthology.orchestra.run.smithi164.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-03-25T16:19:51.628 INFO:teuthology.orchestra.run.smithi164.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-03-25T16:19:51.695 INFO:teuthology.orchestra.run.smithi164.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:19:51.745 INFO:teuthology.orchestra.run.smithi164.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-03-25T16:19:51.796 INFO:teuthology.orchestra.run.smithi164.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T16:19:51.846 INFO:teuthology.orchestra.run.smithi164.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:19:51.896 INFO:teuthology.orchestra.run.smithi164.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T16:19:51.947 INFO:teuthology.orchestra.run.smithi164.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T16:19:52.064 INFO:teuthology.orchestra.run.smithi164.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2024-03-25T16:19:52.156 INFO:teuthology.orchestra.run.smithi164.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-25T16:19:52.215 INFO:teuthology.orchestra.run.smithi164.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-25T16:19:52.273 INFO:teuthology.orchestra.run.smithi164.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-25T16:19:55.807 INFO:teuthology.orchestra.run.smithi164.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 416 kB/s | 4.7 MB 00:11 2024-03-25T16:20:16.029 INFO:teuthology.orchestra.run.smithi164.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 622 kB/s | 17 MB 00:27 2024-03-25T16:20:16.033 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:16.033 INFO:teuthology.orchestra.run.smithi164.stdout:Total 929 kB/s | 30 MB 00:32 2024-03-25T16:20:16.057 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:16.099 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:16.099 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:16.514 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:16.514 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:17.855 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:17.923 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:20:17.957 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:20:17.992 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:20:18.063 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:20:18.108 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:20:18.141 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:20:18.234 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:20:18.320 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:20:18.369 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:20:18.443 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:20:18.495 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:20:18.533 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:20:18.591 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:20:18.646 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:20:18.841 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:20:18.887 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:20:18.926 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:20:18.961 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:20:18.991 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:20:19.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:20:19.101 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:20:19.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:20:19.367 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:20:19.402 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:20:19.445 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:20:19.534 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:20:19.576 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:20:19.648 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:20:19.784 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:20:19.901 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:20:20.060 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:20:20.090 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:20:20.115 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:20:20.115 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:20.115 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:20:20.116 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:20:20.116 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:20:20.116 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:21.557 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:20:21.583 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:22.024 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:20:22.047 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:20:22.047 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:22.047 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:20:22.047 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:20:22.047 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:20:22.048 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:20:22.281 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:20:22.281 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:22.281 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:20:22.281 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:20:22.281 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:20:22.282 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:20:23.973 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:20:23.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:20:23.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:20:23.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:20:23.974 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:20:23.975 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:20:23.976 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:20:23.976 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:20:23.976 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:20:23.976 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:20:23.977 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:20:23.977 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:20:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:20:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:24.872 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:25.250 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base 2024-03-25T16:20:25.820 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:25.917 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:20:25.968 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:25.969 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:20:25.969 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:26.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install cephadm 2024-03-25T16:20:26.598 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:26.747 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:26.748 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:26.748 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:20:26.748 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:26.748 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 747 k 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 754 k 2024-03-25T16:20:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:20:28.011 INFO:teuthology.orchestra.run.smithi164.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 593 kB/s | 747 kB 00:01 2024-03-25T16:20:28.011 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:28.011 INFO:teuthology.orchestra.run.smithi164.stdout:Total 592 kB/s | 747 kB 00:01 2024-03-25T16:20:28.012 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:28.017 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:28.017 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:28.023 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:28.023 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:28.098 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:28.511 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:20:28.599 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:20:29.012 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:29.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:20:30.143 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:30.292 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 145 k 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 436 k 2024-03-25T16:20:30.293 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:20:30.959 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 219 kB/s | 145 kB 00:00 2024-03-25T16:20:30.959 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:30.960 INFO:teuthology.orchestra.run.smithi164.stdout:Total 218 kB/s | 145 kB 00:00 2024-03-25T16:20:30.960 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:30.968 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:30.968 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:31.020 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:31.020 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:31.169 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:31.226 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:20:31.251 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:20:31.251 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:31.251 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:20:31.251 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:32.224 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:20:32.225 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:32.225 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:20:32.225 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:32.225 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:32.225 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:32.487 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr 2024-03-25T16:20:33.056 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:33.156 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:20:33.209 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:33.210 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:20:33.210 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:33.267 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:20:33.829 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:33.978 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:20:33.979 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:33.980 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 6.9 M 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2024-03-25T16:20:33.981 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:20:34.647 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:20:34.847 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T16:20:34.864 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 48 kB/s | 31 kB 00:00 2024-03-25T16:20:34.897 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.6 MB/s | 238 kB 00:00 2024-03-25T16:20:35.322 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:20:35.381 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-25T16:20:35.456 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.4 MB/s | 863 kB 00:00 2024-03-25T16:20:35.715 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 559 kB/s | 144 kB 00:00 2024-03-25T16:20:35.774 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 529 kB/s | 31 kB 00:00 2024-03-25T16:20:35.816 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.7 MB/s | 2.0 MB 00:00 2024-03-25T16:20:35.874 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T16:20:36.308 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 867 kB/s | 1.8 MB 00:02 2024-03-25T16:20:36.312 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:36.312 INFO:teuthology.orchestra.run.smithi164.stdout:Total 2.9 MB/s | 6.9 MB 00:02 2024-03-25T16:20:36.331 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:36.348 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:36.348 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:36.533 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:36.533 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:37.055 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:37.142 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:20:37.177 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:20:37.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:20:37.254 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:20:37.367 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:20:37.505 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:20:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:20:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:20:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:20:37.702 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:20:37.943 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:20:37.986 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:20:38.823 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:20:38.823 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:20:38.823 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:20:38.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:20:39.157 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:20:39.157 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:39.157 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:20:39.157 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:20:39.157 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:39.158 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:39.403 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:20:39.962 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:21 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:40.113 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:20:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-03-25T16:20:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:40.116 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 37 M 2024-03-25T16:20:40.116 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 209 M 2024-03-25T16:20:40.116 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:20:40.490 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-03-25T16:20:40.549 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 487 kB/s | 17 kB 00:00 2024-03-25T16:20:40.590 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-25T16:20:40.749 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:20:40.891 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T16:20:40.942 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T16:20:40.967 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:20:41.150 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-25T16:20:41.309 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T16:20:41.334 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T16:20:41.468 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-25T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.1 MB/s | 7.4 MB 00:06 2024-03-25T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:46.979 INFO:teuthology.orchestra.run.smithi164.stdout:Total 5.4 MB/s | 37 MB 00:06 2024-03-25T16:20:47.023 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:47.048 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:47.048 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:47.427 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:47.428 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:48.141 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:48.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:20:48.263 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:20:48.358 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:20:48.392 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:20:48.571 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:20:48.599 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:20:48.686 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:20:49.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:20:49.478 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:20:50.716 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:20:51.758 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:20:51.791 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:20:54.562 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:20:54.562 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:20:54.563 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:20:54.999 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:20:54.999 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:54.999 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:20:54.999 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:20:54.999 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:55.000 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:20:55.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:20:55.964 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:37 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:20:56.114 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:20:56.115 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:20:56.116 INFO:teuthology.orchestra.run.smithi164.stdout:Install 13 Packages 2024-03-25T16:20:56.117 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:20:56.117 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 2.0 M 2024-03-25T16:20:56.117 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2024-03-25T16:20:56.117 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:20:56.292 INFO:teuthology.orchestra.run.smithi164.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2024-03-25T16:20:56.342 INFO:teuthology.orchestra.run.smithi164.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-25T16:20:56.393 INFO:teuthology.orchestra.run.smithi164.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:20:56.418 INFO:teuthology.orchestra.run.smithi164.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:20:56.443 INFO:teuthology.orchestra.run.smithi164.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi164.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:20:56.669 INFO:teuthology.orchestra.run.smithi164.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-03-25T16:20:56.711 INFO:teuthology.orchestra.run.smithi164.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-25T16:20:56.861 INFO:teuthology.orchestra.run.smithi164.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 77 kB/s | 52 kB 00:00 2024-03-25T16:20:56.886 INFO:teuthology.orchestra.run.smithi164.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 932 kB/s | 203 kB 00:00 2024-03-25T16:20:56.970 INFO:teuthology.orchestra.run.smithi164.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-25T16:20:57.012 INFO:teuthology.orchestra.run.smithi164.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-03-25T16:20:57.129 INFO:teuthology.orchestra.run.smithi164.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 221 kB/s | 59 kB 00:00 2024-03-25T16:20:57.133 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:57.133 INFO:teuthology.orchestra.run.smithi164.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T16:20:57.133 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:20:57.146 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:20:57.146 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:20:57.237 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:20:57.238 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:20:57.745 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:20:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:20:57.972 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:20:58.036 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:20:58.073 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:20:58.123 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:20:58.225 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:20:58.341 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:20:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:20:59.135 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:20:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:20:59.213 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:20:59.262 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:20:59.284 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:00.072 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:00.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:21:00.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:21:00.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:21:00.073 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:21:00.447 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:21:00.447 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:00.447 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:21:00.448 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:21:00.449 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:21:00.449 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:00.449 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:00.701 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:21:01.266 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:01.416 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:01.416 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:21:01.416 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:01.417 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 6.8 M 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 33 M 2024-03-25T16:21:01.418 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-25T16:21:02.190 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:21:02.357 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 347 kB/s | 58 kB 00:00 2024-03-25T16:21:02.382 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:21:02.433 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 679 kB/s | 51 kB 00:00 2024-03-25T16:21:02.641 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 185 kB/s | 138 kB 00:00 2024-03-25T16:21:02.643 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:02.643 INFO:teuthology.orchestra.run.smithi164.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-25T16:21:02.657 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:02.667 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:02.667 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:02.760 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:02.761 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:03.049 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:03.157 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] gperftools-libs-2.9.1-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-2.el9.x86_64.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-25T16:21:03.214 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:21:03.277 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:21:03.794 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:21:03.878 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:21:03.949 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:21:03.972 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:04.800 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:04.800 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:21:04.800 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:21:04.800 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:21:04.801 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:21:04.801 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:21:05.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:21:05.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:21:05.091 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:05.091 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:05.326 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse 2024-03-25T16:21:05.897 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:47 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:06.048 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:06.050 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:06.050 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:21:06.050 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:06.050 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:06.050 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:06.051 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-03-25T16:21:06.052 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:06.052 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 964 k 2024-03-25T16:21:06.052 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.8 M 2024-03-25T16:21:06.052 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:06.179 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 611 kB/s | 79 kB 00:00 2024-03-25T16:21:06.682 INFO:teuthology.orchestra.run.smithi046.stdout:(31/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 kB/s | 309 kB 02:13 2024-03-25T16:21:06.937 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-25T16:21:07.423 INFO:teuthology.orchestra.run.smithi046.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 34 kB/s | 4.4 MB 02:12 2024-03-25T16:21:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:07.425 INFO:teuthology.orchestra.run.smithi046.stdout:Total 416 kB/s | 58 MB 02:22 2024-03-25T16:21:07.457 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:21:07.499 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:21:07.499 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:21:07.979 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 459 kB/s | 885 kB 00:01 2024-03-25T16:21:07.981 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:07.981 INFO:teuthology.orchestra.run.smithi164.stdout:Total 499 kB/s | 964 kB 00:01 2024-03-25T16:21:07.982 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:07.985 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:21:07.986 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:21:08.005 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:08.005 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:08.083 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:08.083 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:08.263 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:08.413 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:21:08.480 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:09.171 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:09.171 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:21:09.490 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:21:09.491 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:09.491 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:09.491 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:21:09.491 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:09.491 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:09.522 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:21:09.622 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:21:09.653 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:21:09.678 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:21:09.710 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:09.725 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:21:09.776 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-volume 2024-03-25T16:21:09.896 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:09.949 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:21:10.333 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:21:10.359 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:10.363 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:10.438 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:10.514 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:10.515 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:10.516 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-03-25T16:21:10.516 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:10.516 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 332 k 2024-03-25T16:21:10.516 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.5 M 2024-03-25T16:21:10.516 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:10.527 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:21:10.617 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 684 kB/s | 69 kB 00:00 2024-03-25T16:21:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:10.719 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:21:10.748 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:21:10.792 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:21:10.861 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:10.884 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:10.946 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:21:10.973 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:21:11.084 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:21:11.125 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:21:11.165 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:21:11.198 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:21:11.227 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:21:11.259 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:21:11.367 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 310 kB/s | 264 kB 00:00 2024-03-25T16:21:11.368 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:11.369 INFO:teuthology.orchestra.run.smithi164.stdout:Total 390 kB/s | 332 kB 00:00 2024-03-25T16:21:11.369 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:11.378 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:11.378 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:11.419 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:11.419 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:11.574 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:11.583 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:21:11.614 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:11.705 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:21:11.735 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:21:11.751 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:21:11.776 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:21:11.795 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:21:11.811 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:21:11.811 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:11.811 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:21:11.811 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:21:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:21:12.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:21:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:21:12.593 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:21:12.925 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:13.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel 2024-03-25T16:21:13.892 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:55 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:14.040 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:21:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:14.042 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 127 k 2024-03-25T16:21:14.042 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 456 k 2024-03-25T16:21:14.042 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:14.655 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:14.690 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:14.716 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 188 kB/s | 127 kB 00:00 2024-03-25T16:21:14.716 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:14.716 INFO:teuthology.orchestra.run.smithi164.stdout:Total 188 kB/s | 127 kB 00:00 2024-03-25T16:21:14.717 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:14.722 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:14.722 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:14.777 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:14.928 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:15.030 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:15.268 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:21:15.302 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:15.386 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:21:15.386 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:21:15.386 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:21:15.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:15.772 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:15.773 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:15.773 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:15.773 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:15.773 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:15.773 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:15.993 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2 2024-03-25T16:21:16.569 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:58 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:16.668 INFO:teuthology.orchestra.run.smithi164.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:16.721 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:16.722 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:16.723 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:16.788 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs-devel 2024-03-25T16:21:17.356 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:17.505 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:17.505 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 31 k 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 140 k 2024-03-25T16:21:17.506 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:17.952 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 70 kB/s | 31 kB 00:00 2024-03-25T16:21:17.953 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:17.953 INFO:teuthology.orchestra.run.smithi164.stdout:Total 70 kB/s | 31 kB 00:00 2024-03-25T16:21:17.953 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:17.956 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:17.956 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:17.994 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:17.994 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:18.251 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:18.667 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:18.925 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:19.108 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2 2024-03-25T16:21:19.677 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:19.774 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:19.827 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:19.828 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:19.828 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:19.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1 2024-03-25T16:21:20.450 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:20.549 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:20.600 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:20.602 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:20.602 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:20.655 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados 2024-03-25T16:21:21.220 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:03 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:21.316 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:21.367 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:21.368 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:21.368 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:21.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw 2024-03-25T16:21:21.994 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:03 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:22.090 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:22.142 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:22.143 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:22.143 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:22.199 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs 2024-03-25T16:21:22.774 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:04 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:22.874 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:22.929 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:22.930 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:22.930 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:22.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd 2024-03-25T16:21:23.573 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:05 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:23.671 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:23.723 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:23.724 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T16:21:23.724 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:23.782 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse 2024-03-25T16:21:24.349 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:24.498 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:24.499 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:24.499 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-25T16:21:24.499 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:24.499 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:24.499 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 85 k 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 231 k 2024-03-25T16:21:24.500 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:25.122 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:25.122 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-03-25T16:21:25.122 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-03-25T16:21:25.122 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-03-25T16:21:25.123 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-03-25T16:21:25.123 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-03-25T16:21:25.123 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-03-25T16:21:25.123 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-03-25T16:21:25.123 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:25.272 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 110 kB/s | 85 kB 00:00 2024-03-25T16:21:25.273 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:25.273 INFO:teuthology.orchestra.run.smithi164.stdout:Total 110 kB/s | 85 kB 00:00 2024-03-25T16:21:25.273 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:25.281 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:25.281 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:25.337 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:25.337 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:25.481 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:25.587 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:26.034 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:26.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:26.094 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:21:26.094 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:26.094 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:21:26.095 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:26.095 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:21:26.095 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:26.096 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:26.173 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:21:26.173 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:26.369 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:26.369 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:26.370 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:26.370 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:26.370 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:26.370 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:26.598 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror 2024-03-25T16:21:27.175 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:09 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:27.329 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:27.329 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.1 M 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 13 M 2024-03-25T16:21:27.330 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:27.345 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:27.346 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:21:27.347 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:21:27.348 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:21:27.349 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:21:27.349 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:21:27.349 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:21:27.349 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:21:28.480 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:21:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:21:28.482 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:28.484 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:21:28.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test 2024-03-25T16:21:29.420 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:47 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:21:29.574 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:29.575 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 48 M 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 204 M 2024-03-25T16:21:29.576 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:21:29.767 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-25T16:21:29.867 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T16:21:29.893 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T16:21:30.018 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:21:30.543 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 181 kB/s | 163 kB 00:00 2024-03-25T16:21:30.708 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 936 kB/s | 3.1 MB 00:03 2024-03-25T16:21:30.709 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:30.709 INFO:teuthology.orchestra.run.smithi164.stdout:Total 935 kB/s | 3.1 MB 00:03 2024-03-25T16:21:30.710 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:30.721 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:30.721 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:30.797 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:30.798 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:31.227 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:31.292 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:21:31.317 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:32.462 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:32.719 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd 2024-03-25T16:21:33.304 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:33.454 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:33.455 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:21:33.456 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:33.456 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 171 k 2024-03-25T16:21:33.456 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 494 k 2024-03-25T16:21:33.456 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:21:34.179 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 237 kB/s | 171 kB 00:00 2024-03-25T16:21:34.259 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:34.259 INFO:teuthology.orchestra.run.smithi164.stdout:Total 237 kB/s | 171 kB 00:00 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:21:34.391 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:21:34.485 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:34.938 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:35.234 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:35.235 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:35.235 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:21:35.235 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:35.235 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:35.235 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:21:35.473 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:21:36.043 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:21:36.197 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-03-25T16:21:36.198 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:21:36.199 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 91 M 2024-03-25T16:21:36.199 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 498 M 2024-03-25T16:21:36.199 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:22:00.605 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 942 kB/s | 22 MB 00:24 2024-03-25T16:22:30.530 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 795 kB/s | 47 MB 01:00 2024-03-25T16:22:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:Total 809 kB/s | 48 MB 01:00 2024-03-25T16:22:30.544 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:22:30.566 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:22:30.566 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:22:30.957 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:22:30.957 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:22:31.595 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:22:31.672 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:22:31.721 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:22:31.760 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:31.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:22:31.840 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:36.539 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:36.617 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:39.005 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:39.006 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:22:39.006 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:22:39.006 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:39.006 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:22:39.006 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:22:39.353 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:22:39.354 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:22:39.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph 2024-03-25T16:22:40.218 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:22:40.383 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:22:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================= 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================= 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:22:40.386 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:22:40.387 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:22:40.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:22:40.389 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================= 2024-03-25T16:22:40.390 INFO:teuthology.orchestra.run.smithi046.stdout:Install 36 Packages 2024-03-25T16:22:40.390 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:22:40.390 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 30 M 2024-03-25T16:22:40.390 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 107 M 2024-03-25T16:22:40.391 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:22:41.417 INFO:teuthology.orchestra.run.smithi046.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:22:42.876 INFO:teuthology.orchestra.run.smithi046.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 838 kB/s | 1.5 MB 00:01 2024-03-25T16:22:43.409 INFO:teuthology.orchestra.run.smithi046.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 924 kB/s | 2.1 MB 00:02 2024-03-25T16:22:43.618 INFO:teuthology.orchestra.run.smithi046.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.2 MB/s | 247 kB 00:00 2024-03-25T16:22:43.744 INFO:teuthology.orchestra.run.smithi046.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:22:43.785 INFO:teuthology.orchestra.run.smithi046.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:22:43.810 INFO:teuthology.orchestra.run.smithi046.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:22:43.836 INFO:teuthology.orchestra.run.smithi046.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T16:22:43.861 INFO:teuthology.orchestra.run.smithi046.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:22:43.928 INFO:teuthology.orchestra.run.smithi046.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-25T16:22:44.094 INFO:teuthology.orchestra.run.smithi046.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-03-25T16:22:44.136 INFO:teuthology.orchestra.run.smithi046.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:22:44.170 INFO:teuthology.orchestra.run.smithi046.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:22:44.212 INFO:teuthology.orchestra.run.smithi046.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:22:44.279 INFO:teuthology.orchestra.run.smithi046.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-25T16:22:44.421 INFO:teuthology.orchestra.run.smithi046.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 4.7 MB 00:03 2024-03-25T16:22:44.538 INFO:teuthology.orchestra.run.smithi046.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-25T16:22:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 202 kB/s | 43 kB 00:00 2024-03-25T16:22:44.781 INFO:teuthology.orchestra.run.smithi046.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 421 kB/s | 151 kB 00:00 2024-03-25T16:22:44.847 INFO:teuthology.orchestra.run.smithi046.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-25T16:22:44.915 INFO:teuthology.orchestra.run.smithi046.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T16:22:44.940 INFO:teuthology.orchestra.run.smithi046.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-25T16:22:44.990 INFO:teuthology.orchestra.run.smithi046.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T16:22:45.015 INFO:teuthology.orchestra.run.smithi046.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-25T16:22:45.057 INFO:teuthology.orchestra.run.smithi046.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T16:22:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-25T16:22:45.124 INFO:teuthology.orchestra.run.smithi046.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-25T16:22:45.150 INFO:teuthology.orchestra.run.smithi046.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T16:22:45.208 INFO:teuthology.orchestra.run.smithi046.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T16:22:45.234 INFO:teuthology.orchestra.run.smithi046.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-25T16:22:45.275 INFO:teuthology.orchestra.run.smithi046.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T16:22:45.301 INFO:teuthology.orchestra.run.smithi046.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-03-25T16:22:45.351 INFO:teuthology.orchestra.run.smithi046.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T16:22:45.393 INFO:teuthology.orchestra.run.smithi046.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T16:22:45.418 INFO:teuthology.orchestra.run.smithi046.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 301 kB/s | 20 kB 00:00 2024-03-25T16:23:05.900 INFO:teuthology.orchestra.run.smithi046.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 746 kB/s | 17 MB 00:23 2024-03-25T16:23:05.905 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:05.905 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.2 MB/s | 30 MB 00:25 2024-03-25T16:23:05.928 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:05.975 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:05.975 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:06.393 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:06.393 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:07.754 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:07.819 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:23:07.860 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:23:07.897 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:23:07.970 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:23:08.004 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:23:08.037 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:23:08.133 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:23:08.208 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:23:08.256 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:23:08.329 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:23:08.382 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:23:08.420 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:23:08.486 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:23:08.542 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:23:08.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:23:08.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:23:08.830 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:23:08.864 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:23:08.894 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:23:08.973 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:23:09.005 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:09.038 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:23:09.078 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:23:09.113 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:23:09.148 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:23:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:23:09.279 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:23:09.353 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:23:09.493 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:23:09.626 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:09.803 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:23:09.881 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:09.905 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:23:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:23:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:09.906 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:23:09.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:11.435 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:11.462 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:23:11.463 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:11.463 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:23:11.463 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:11.463 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:23:11.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:23:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:23:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:23:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:23:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:23:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:23:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:23:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:23:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:23:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:23:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:23:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:23:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:23:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:23:14.970 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 706 kB/s | 68 MB 01:38 2024-03-25T16:23:14.971 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:14.972 INFO:teuthology.orchestra.run.smithi164.stdout:Total 939 kB/s | 91 MB 01:38 2024-03-25T16:23:14.972 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:23:14.976 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:23:14.976 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.124 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:23:15.125 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:23:15.126 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:15.127 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:15.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base 2024-03-25T16:23:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:23:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:23:16.145 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:16.245 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:16.299 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:16.300 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:23:16.300 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:16.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install cephadm 2024-03-25T16:23:16.939 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:23:17.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:17.097 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 747 k 2024-03-25T16:23:17.097 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 754 k 2024-03-25T16:23:17.097 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:23:18.658 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:23:18.711 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 463 kB/s | 747 kB 00:01 2024-03-25T16:23:18.712 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:18.712 INFO:teuthology.orchestra.run.smithi046.stdout:Total 462 kB/s | 747 kB 00:01 2024-03-25T16:23:18.713 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:18.719 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:18.719 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:18.726 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:18.726 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:18.807 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:19.283 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:19.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:19.903 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:20.214 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:20.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:23:21.013 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:21.169 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:23:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:23:21.171 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:23:21.171 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:21.171 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 145 k 2024-03-25T16:23:21.171 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 436 k 2024-03-25T16:23:21.171 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:23:21.876 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 206 kB/s | 145 kB 00:00 2024-03-25T16:23:21.877 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:21.877 INFO:teuthology.orchestra.run.smithi046.stdout:Total 205 kB/s | 145 kB 00:00 2024-03-25T16:23:21.877 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:21.886 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:21.886 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:21.943 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:21.943 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:22.088 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:22.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:22.168 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:22.168 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:22.168 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:23:22.168 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:23.113 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:23.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr 2024-03-25T16:23:24.092 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:24.192 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:23:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:24.316 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:23:24.885 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:25.042 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:23:25.043 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================== 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-03-25T16:23:25.044 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:25.045 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 6.9 M 2024-03-25T16:23:25.045 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 26 M 2024-03-25T16:23:25.045 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:23:25.764 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:23:25.847 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 61 kB/s | 31 kB 00:00 2024-03-25T16:23:25.853 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:23:25.914 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T16:23:25.939 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-25T16:23:26.148 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T16:23:26.390 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-25T16:23:26.449 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T16:23:26.557 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T16:23:26.666 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T16:23:26.766 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T16:23:27.425 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.0 MB/s | 2.0 MB 00:01 2024-03-25T16:23:28.633 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 553 kB/s | 1.8 MB 00:03 2024-03-25T16:23:28.637 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:28.637 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.9 MB/s | 6.9 MB 00:03 2024-03-25T16:23:28.657 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:28.678 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:28.678 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:28.862 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:28.862 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:29.178 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:23:29.178 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:23:29.377 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:23:29.441 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:23:29.451 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:23:29.486 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:23:29.533 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:23:29.571 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:23:29.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:23:29.681 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:23:29.819 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:23:29.910 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:23:29.960 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:23:29.996 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:30.028 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:23:30.228 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:23:30.299 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:23:30.327 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:30.389 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:23:30.389 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:23:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:23:30.389 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 416 M 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.7 G 2024-03-25T16:23:30.390 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:23:31.093 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:23:31.093 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:23:31.093 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:23:31.094 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:23:31.446 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:23:31.446 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:31.446 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:23:31.446 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:23:31.446 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:31.447 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:31.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:23:32.251 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:32.408 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:23:32.410 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-03-25T16:23:32.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:32.412 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 37 M 2024-03-25T16:23:32.412 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 209 M 2024-03-25T16:23:32.412 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:23:32.890 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-03-25T16:23:32.923 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 526 kB/s | 17 kB 00:00 2024-03-25T16:23:32.982 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-25T16:23:33.007 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 34 kB 00:00 2024-03-25T16:23:33.191 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:23:33.224 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-25T16:23:33.249 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:23:33.567 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T16:23:33.633 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-25T16:23:33.767 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-03-25T16:23:33.876 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-25T16:23:46.308 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 558 kB/s | 7.4 MB 00:13 2024-03-25T16:23:46.310 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:46.310 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.7 MB/s | 37 MB 00:13 2024-03-25T16:23:46.355 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:46.380 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:46.380 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:46.766 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:46.766 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:47.494 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:47.573 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:23:47.604 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:23:47.702 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:47.757 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:23:47.936 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:47.965 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:48.050 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:23:48.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:23:48.864 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:23:50.173 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:23:51.271 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:23:51.297 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:53.719 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:23:53.720 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:23:54.137 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:54.138 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:23:54.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:23:55.069 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:23:55.226 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:23:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:23:55.227 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:23:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:23:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:23:55.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:23:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 2.0 M 2024-03-25T16:23:55.230 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 26 M 2024-03-25T16:23:55.230 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:23:55.457 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-25T16:23:55.483 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-25T16:23:55.558 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:23:55.583 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-25T16:23:55.608 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:23:55.625 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:23:55.876 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 121 kB/s | 32 kB 00:00 2024-03-25T16:23:55.892 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-25T16:23:55.935 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 85 kB/s | 52 kB 00:00 2024-03-25T16:23:56.210 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 607 kB/s | 203 kB 00:00 2024-03-25T16:23:56.269 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2024-03-25T16:23:56.394 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-03-25T16:23:56.969 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 971 kB/s | 1.0 MB 00:01 2024-03-25T16:23:56.971 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:56.971 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-25T16:23:56.972 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:23:56.987 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:23:56.988 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:23:57.091 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:23:57.091 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:23:57.753 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:23:57.923 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:23:57.988 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:23:58.044 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:23:58.096 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:23:58.139 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:23:58.238 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:23:58.344 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:23:58.399 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:23:59.155 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:59.211 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:59.245 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:59.294 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:59.323 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:24:00.058 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:24:00.059 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:24:00.535 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:24:00.535 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:00.535 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:24:00.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:00.537 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:00.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:24:01.388 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:01.546 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:24:01.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 6.8 M 2024-03-25T16:24:01.548 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 33 M 2024-03-25T16:24:01.549 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:02.029 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T16:24:02.054 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 684 kB/s | 248 kB 00:00 2024-03-25T16:24:02.538 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 163 kB/s | 138 kB 00:00 2024-03-25T16:24:02.580 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 110 kB/s | 58 kB 00:00 2024-03-25T16:24:02.864 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 657 kB/s | 548 kB 00:00 2024-03-25T16:24:03.080 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 94 kB/s | 51 kB 00:00 2024-03-25T16:24:03.085 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:03.085 INFO:teuthology.orchestra.run.smithi046.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-03-25T16:24:03.099 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:03.202 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:03.502 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:03.670 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:24:03.724 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:24:04.278 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:24:04.364 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:24:04.431 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:24:04.461 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:24:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:24:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:24:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:24:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:24:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:24:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:24:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:24:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:24:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:24:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:05.536 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:05.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse 2024-03-25T16:24:06.341 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:06.497 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:06.498 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-03-25T16:24:06.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:06.499 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 964 k 2024-03-25T16:24:06.499 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.8 M 2024-03-25T16:24:06.499 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:06.674 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 450 kB/s | 79 kB 00:00 2024-03-25T16:24:08.157 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 534 kB/s | 885 kB 00:01 2024-03-25T16:24:08.159 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:08.159 INFO:teuthology.orchestra.run.smithi046.stdout:Total 580 kB/s | 964 kB 00:01 2024-03-25T16:24:08.159 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:08.183 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:08.184 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:08.263 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:08.264 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:08.448 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:08.580 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:24:08.659 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:09.336 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:09.336 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:09.634 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:09.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-volume 2024-03-25T16:24:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:10.646 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:10.646 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-03-25T16:24:10.647 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:10.648 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 332 k 2024-03-25T16:24:10.648 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.5 M 2024-03-25T16:24:10.648 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:10.775 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 542 kB/s | 69 kB 00:00 2024-03-25T16:24:11.846 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 220 kB/s | 264 kB 00:01 2024-03-25T16:24:11.847 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:11.847 INFO:teuthology.orchestra.run.smithi046.stdout:Total 277 kB/s | 332 kB 00:01 2024-03-25T16:24:11.848 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:11.858 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:11.858 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:11.904 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:11.905 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:24:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:24:12.314 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:12.314 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:24:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:13.516 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:13.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel 2024-03-25T16:24:14.462 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:14.618 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:14.618 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 127 k 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 456 k 2024-03-25T16:24:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:15.395 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 163 kB/s | 127 kB 00:00 2024-03-25T16:24:15.395 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:15.395 INFO:teuthology.orchestra.run.smithi046.stdout:Total 163 kB/s | 127 kB 00:00 2024-03-25T16:24:15.396 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:15.465 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:15.465 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:15.616 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:15.722 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:16.165 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:16.416 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:16.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:16.417 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:16.417 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:16.417 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:16.417 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:16.682 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2 2024-03-25T16:24:17.263 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:17.368 INFO:teuthology.orchestra.run.smithi046.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:17.424 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:17.425 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:17.425 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:17.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs-devel 2024-03-25T16:24:18.089 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:18.246 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 31 k 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 140 k 2024-03-25T16:24:18.247 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:18.807 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T16:24:18.808 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:18.808 INFO:teuthology.orchestra.run.smithi046.stdout:Total 55 kB/s | 31 kB 00:00 2024-03-25T16:24:18.808 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:18.812 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:18.812 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:18.836 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:18.836 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:18.970 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:19.083 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.465 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.722 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:19.723 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:19.723 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:19.723 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:19.723 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:19.723 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:19.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2 2024-03-25T16:24:20.480 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:20.584 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:20.638 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:20.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1 2024-03-25T16:24:21.275 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:21.376 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:21.430 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:21.431 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:21.431 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:21.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados 2024-03-25T16:24:22.067 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:22.169 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:22.227 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:22.228 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:22.228 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:22.283 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw 2024-03-25T16:24:22.867 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:22.967 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:23.022 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:23.075 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs 2024-03-25T16:24:23.650 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:23.750 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:23.806 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:23.807 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:23.807 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:23.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd 2024-03-25T16:24:24.442 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:24:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T16:24:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:24.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse 2024-03-25T16:24:25.245 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:25.400 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:25.451 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:25.451 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:25.451 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:25.451 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:25.451 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 85 k 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 231 k 2024-03-25T16:24:25.452 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:26.111 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 85 kB 00:00 2024-03-25T16:24:26.112 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:26.112 INFO:teuthology.orchestra.run.smithi046.stdout:Total 120 kB/s | 85 kB 00:00 2024-03-25T16:24:26.112 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:26.120 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:26.120 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:26.179 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:26.180 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:26.321 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:26.412 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:26.798 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:27.073 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:27.296 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror 2024-03-25T16:24:27.872 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:28.028 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:28.029 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:28.029 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:24:28.029 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.1 M 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 13 M 2024-03-25T16:24:28.030 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:36.122 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 391 kB/s | 3.1 MB 00:08 2024-03-25T16:24:36.123 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:36.123 INFO:teuthology.orchestra.run.smithi046.stdout:Total 391 kB/s | 3.1 MB 00:08 2024-03-25T16:24:36.123 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:36.136 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:36.136 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:36.215 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:36.216 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:36.666 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:36.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:36.738 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:36.739 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:36.739 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:24:36.739 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:36.739 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:36.739 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:37.804 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:38.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd 2024-03-25T16:24:38.644 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:38.799 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 171 k 2024-03-25T16:24:38.800 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 494 k 2024-03-25T16:24:38.801 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:24:39.709 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 189 kB/s | 171 kB 00:00 2024-03-25T16:24:39.709 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:39.709 INFO:teuthology.orchestra.run.smithi046.stdout:Total 189 kB/s | 171 kB 00:00 2024-03-25T16:24:39.710 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:24:39.720 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:24:39.720 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:24:39.782 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:24:39.783 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:24:39.937 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:24:40.043 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:40.516 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:40.798 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:40.799 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:40.799 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:24:40.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:40.799 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:40.799 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:24:41.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:24:41.574 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:24:41.734 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:24:41.734 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:41.734 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:24:41.734 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:41.734 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-03-25T16:24:41.735 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:24:41.736 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 91 M 2024-03-25T16:24:41.736 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 498 M 2024-03-25T16:24:41.736 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:25:24.417 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 539 kB/s | 22 MB 00:42 2024-03-25T16:27:24.804 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 428 kB/s | 68 MB 02:43 2024-03-25T16:27:24.805 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:24.805 INFO:teuthology.orchestra.run.smithi046.stdout:Total 569 kB/s | 91 MB 02:43 2024-03-25T16:27:24.806 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:27:24.810 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:27:24.810 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:27:25.419 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:27:25.420 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:27:28.691 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:27:36.200 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:36.274 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:39.518 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:39.518 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:27:39.790 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:39.790 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:27:39.791 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:27:39.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:39.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:39.791 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:27:39.791 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:27:40.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:27:40.587 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:08:58 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:27:40.746 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:27:40.749 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 416 M 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.7 G 2024-03-25T16:27:40.750 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:46:51.250 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 304 kB/s | 416 MB 23:20 2024-03-25T16:46:51.348 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:46:51.349 INFO:teuthology.orchestra.run.smithi164.stdout:Total 304 kB/s | 416 MB 23:20 2024-03-25T16:46:51.349 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:46:51.350 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:46:51.350 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:46:53.720 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:46:53.720 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:47:35.906 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:47:36.002 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:40.575 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:40.824 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:47:41.040 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:47:41.609 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:29:23 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:47:41.758 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:47:41.759 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:47:41.759 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:47:41.759 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 1.4 M 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 6.4 M 2024-03-25T16:47:41.760 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:47:45.914 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 352 kB/s | 1.4 MB 00:04 2024-03-25T16:47:45.914 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:Total 352 kB/s | 1.4 MB 00:04 2024-03-25T16:47:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:47:45.919 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:47:45.919 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:47:45.964 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:47:45.964 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:47:46.216 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:47:46.329 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:46.796 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:47.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:47.060 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:47.060 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:47:47.060 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:47.060 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:47.060 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:47:47.231 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:47:47.796 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:29:29 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:47:47.943 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:47:47.944 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:47:47.945 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:47:47.945 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 176 M 2024-03-25T16:47:47.945 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 753 M 2024-03-25T16:47:47.945 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T16:56:02.425 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 250 kB/s | 416 MB 28:21 2024-03-25T16:56:02.425 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:02.425 INFO:teuthology.orchestra.run.smithi046.stdout:Total 250 kB/s | 416 MB 28:21 2024-03-25T16:56:02.425 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:56:02.428 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:56:02.429 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:56:05.018 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:56:05.018 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:56:49.622 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:56:49.722 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:54.167 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 329 kB/s | 176 MB 09:06 2024-03-25T16:56:54.167 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:54.168 INFO:teuthology.orchestra.run.smithi164.stdout:Total 329 kB/s | 176 MB 09:06 2024-03-25T16:56:54.168 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T16:56:54.171 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T16:56:54.171 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T16:56:55.231 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T16:56:55.231 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T16:56:57.500 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:57.761 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:57.761 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:56:57.762 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:56:57.762 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:57.762 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:56:57.762 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:56:58.012 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:56:58.594 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:38:16 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:56:58.748 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:56:58.748 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 1.4 M 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 6.4 M 2024-03-25T16:56:58.749 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:57:01.249 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 585 kB/s | 1.4 MB 00:02 2024-03-25T16:57:01.249 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:01.250 INFO:teuthology.orchestra.run.smithi046.stdout:Total 584 kB/s | 1.4 MB 00:02 2024-03-25T16:57:01.250 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T16:57:01.254 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T16:57:01.254 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T16:57:01.301 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T16:57:01.301 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T16:57:01.568 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T16:57:01.675 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:02.212 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:57:02.499 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T16:57:02.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:57:03.250 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:38:21 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T16:57:03.407 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:57:03.408 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 176 M 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 753 M 2024-03-25T16:57:03.409 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T16:57:13.340 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T16:57:13.490 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:18.183 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:57:18.432 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T16:57:18.629 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:57:19.189 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:39:01 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T16:57:19.338 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T16:57:19.338 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T16:57:19.339 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T16:57:19.340 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 672 M 2024-03-25T16:57:19.340 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 3.1 G 2024-03-25T16:57:19.340 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:04:03.736 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 428 kB/s | 176 MB 07:00 2024-03-25T17:04:03.737 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:03.737 INFO:teuthology.orchestra.run.smithi046.stdout:Total 428 kB/s | 176 MB 07:00 2024-03-25T17:04:03.737 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:04:03.739 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:04:03.739 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:04:04.850 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:04:04.850 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:04:23.932 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:04:24.018 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:28.491 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:04:28.749 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:04:28.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:04:29.522 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:45:47 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:04:29.677 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:04:29.677 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:04:29.677 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 672 M 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 3.1 G 2024-03-25T17:04:29.678 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:29:12.555 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 360 kB/s | 672 MB 31:53 2024-03-25T17:29:12.593 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:29:12.593 INFO:teuthology.orchestra.run.smithi164.stdout:Total 360 kB/s | 672 MB 31:53 2024-03-25T17:29:12.593 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:29:12.594 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:29:12.594 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:29:16.516 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:29:16.516 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:30:27.278 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:30:27.621 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:36.756 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:30:37.017 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:30:37.317 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:30:37.887 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:12:19 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:30:37.984 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:30:38.036 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:30:38.037 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-25T17:30:38.037 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:30:38.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:30:38.676 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:12:20 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:30:38.826 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:30:38.826 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 20 M 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 103 M 2024-03-25T17:30:38.827 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:31:29.251 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 400 kB/s | 20 MB 00:50 2024-03-25T17:31:29.252 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:29.252 INFO:teuthology.orchestra.run.smithi164.stdout:Total 400 kB/s | 20 MB 00:50 2024-03-25T17:31:29.252 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:31:29.256 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:31:29.256 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:31:29.407 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:31:29.407 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:31:31.643 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:31:32.025 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:32.930 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:31:33.192 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:31:33.354 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:31:33.923 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:13:15 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:31:34.070 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:31:34.071 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 48 M 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 173 M 2024-03-25T17:31:34.072 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:34:20.157 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 48 MB 02:46 2024-03-25T17:34:20.157 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:20.157 INFO:teuthology.orchestra.run.smithi164.stdout:Total 294 kB/s | 48 MB 02:46 2024-03-25T17:34:20.158 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:34:20.160 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:34:20.160 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:34:20.475 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:34:20.475 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:34:25.223 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:34:25.361 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:26.729 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:34:26.997 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:34:27.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:34:27.745 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:16:09 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:34:27.894 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:34:27.895 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:34:27.896 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 99 M 2024-03-25T17:34:27.896 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 449 M 2024-03-25T17:34:27.896 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:35:14.137 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 373 kB/s | 672 MB 30:44 2024-03-25T17:35:14.137 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:14.137 INFO:teuthology.orchestra.run.smithi046.stdout:Total 373 kB/s | 672 MB 30:44 2024-03-25T17:35:14.137 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:35:14.141 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:35:14.142 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:35:18.293 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:35:18.293 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:36:33.211 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:36:33.316 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:36:41.116 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:36:41.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:36:41.982 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:17:59 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:36:42.081 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:36:42.136 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:36:42.137 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-03-25T17:36:42.137 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:36:42.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:36:42.789 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:18:00 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:36:42.943 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:36:42.944 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:36:42.944 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:36:42.944 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:36:42.944 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:36:42.944 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 20 M 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 103 M 2024-03-25T17:36:42.945 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:37:42.311 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 340 kB/s | 20 MB 00:59 2024-03-25T17:37:42.313 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:42.313 INFO:teuthology.orchestra.run.smithi046.stdout:Total 340 kB/s | 20 MB 00:59 2024-03-25T17:37:42.314 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:37:42.316 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:37:42.316 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:37:42.472 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:37:42.472 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:37:44.829 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:37:44.938 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:45.942 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:37:46.198 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:37:46.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:37:46.947 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:19:04 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:37:47.101 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:37:47.102 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:37:47.102 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 48 M 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 173 M 2024-03-25T17:37:47.103 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:38:52.113 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 382 kB/s | 99 MB 04:24 2024-03-25T17:38:52.114 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:52.114 INFO:teuthology.orchestra.run.smithi164.stdout:Total 382 kB/s | 99 MB 04:24 2024-03-25T17:38:52.114 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:38:52.117 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:38:52.117 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:38:52.725 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:38:52.725 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:39:02.958 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:39:03.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:05.911 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:39:06.168 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:39:06.357 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:39:06.923 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:20:48 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:39:07.072 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:39:07.073 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 287 M 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.2 G 2024-03-25T17:39:07.074 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:39:47.727 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 405 kB/s | 48 MB 02:00 2024-03-25T17:39:47.728 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:47.728 INFO:teuthology.orchestra.run.smithi046.stdout:Total 405 kB/s | 48 MB 02:00 2024-03-25T17:39:47.728 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:39:47.732 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:39:47.732 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:39:48.059 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:39:48.059 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:39:53.088 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:39:53.178 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:54.518 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:54.776 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:54.776 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:39:54.776 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:39:54.776 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:54.776 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:39:54.777 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:39:54.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:39:55.535 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:21:13 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:39:55.690 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:39:55.691 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 99 M 2024-03-25T17:39:55.692 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 449 M 2024-03-25T17:39:55.692 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:44:03.805 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 407 kB/s | 99 MB 04:08 2024-03-25T17:44:03.806 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:03.806 INFO:teuthology.orchestra.run.smithi046.stdout:Total 407 kB/s | 99 MB 04:08 2024-03-25T17:44:03.806 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:44:03.808 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:44:03.808 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:44:04.446 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:44:04.446 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:44:15.359 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:44:15.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:18.277 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:18.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:18.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:44:18.527 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:44:18.527 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:18.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:44:18.527 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:44:18.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:44:19.294 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:25:37 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:44:19.451 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:44:19.452 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:44:19.452 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:44:19.452 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:44:19.452 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:44:19.452 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 287 M 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.2 G 2024-03-25T17:44:19.453 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:48:58.476 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 497 kB/s | 287 MB 09:51 2024-03-25T17:48:58.476 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:58.476 INFO:teuthology.orchestra.run.smithi164.stdout:Total 497 kB/s | 287 MB 09:51 2024-03-25T17:48:58.477 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:48:58.479 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:48:58.479 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:49:00.200 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:49:00.200 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:49:29.687 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:49:29.788 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:37.137 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:49:37.417 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:49:37.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:49:38.203 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:31:20 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:49:38.351 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:49:38.352 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:49:38.353 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:49:38.353 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 5.9 M 2024-03-25T17:49:38.353 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 28 M 2024-03-25T17:49:38.353 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:50:00.821 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 271 kB/s | 5.9 MB 00:22 2024-03-25T17:50:00.822 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:00.822 INFO:teuthology.orchestra.run.smithi164.stdout:Total 271 kB/s | 5.9 MB 00:22 2024-03-25T17:50:00.823 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:50:00.827 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:50:00.827 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:50:00.900 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:50:00.900 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:50:01.644 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:50:01.753 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:02.315 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:02.599 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:02.599 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:02.599 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:50:02.600 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:02.600 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:02.600 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:50:02.774 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:50:03.342 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:31:45 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:50:03.494 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 109 k 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 440 k 2024-03-25T17:50:03.496 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:50:04.410 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 119 kB/s | 109 kB 00:00 2024-03-25T17:50:04.410 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:04.410 INFO:teuthology.orchestra.run.smithi164.stdout:Total 119 kB/s | 109 kB 00:00 2024-03-25T17:50:04.410 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:50:04.412 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:50:04.412 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:50:04.448 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:50:04.448 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:50:04.568 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:50:04.664 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:50:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:05.306 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:50:05.498 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:50:06.062 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:31:48 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:50:06.211 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:50:06.212 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 5.4 M 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 25 M 2024-03-25T17:50:06.213 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:50:26.700 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 268 kB/s | 5.4 MB 00:20 2024-03-25T17:50:26.701 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:26.701 INFO:teuthology.orchestra.run.smithi164.stdout:Total 267 kB/s | 5.4 MB 00:20 2024-03-25T17:50:26.701 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:50:26.705 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:50:26.705 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:50:26.773 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:50:26.773 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:50:27.442 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:50:27.540 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:50:28.041 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:28.298 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:50:28.522 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2-debuginfo 2024-03-25T17:50:29.088 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:32:11 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:50:29.237 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:50:29.238 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:29.238 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:50:29.238 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:29.238 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 44 M 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 253 M 2024-03-25T17:50:29.239 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:51:51.244 INFO:teuthology.orchestra.run.smithi164.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 553 kB/s | 44 MB 01:22 2024-03-25T17:51:51.245 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:51.246 INFO:teuthology.orchestra.run.smithi164.stdout:Total 553 kB/s | 44 MB 01:22 2024-03-25T17:51:51.246 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:51:51.249 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:51:51.249 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:51:51.550 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:51:51.550 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:51:56.607 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:51:56.692 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:58.476 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:51:58.742 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:51:58.959 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:51:59.525 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:33:41 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:51:59.683 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:51:59.684 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 52 M 2024-03-25T17:51:59.684 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 308 M 2024-03-25T17:51:59.684 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:53:48.967 INFO:teuthology.orchestra.run.smithi164.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 486 kB/s | 52 MB 01:49 2024-03-25T17:53:48.967 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:48.967 INFO:teuthology.orchestra.run.smithi164.stdout:Total 486 kB/s | 52 MB 01:49 2024-03-25T17:53:48.967 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:53:48.970 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:53:48.970 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:53:49.314 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:53:49.315 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:53:55.370 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:53:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:57.669 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:53:57.940 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:53:58.139 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:53:58.700 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:35:40 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:53:58.851 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 615 k 2024-03-25T17:53:58.852 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.1 M 2024-03-25T17:53:58.853 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:54:02.200 INFO:teuthology.orchestra.run.smithi164.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 184 kB/s | 615 kB 00:03 2024-03-25T17:54:02.201 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:02.201 INFO:teuthology.orchestra.run.smithi164.stdout:Total 184 kB/s | 615 kB 00:03 2024-03-25T17:54:02.201 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:54:02.205 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:54:02.205 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:54:02.247 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:54:02.247 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:54:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:54:02.516 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:02.909 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:03.166 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:03.166 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:03.166 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:54:03.167 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:03.167 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:03.167 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:54:03.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:54:03.939 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:35:45 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:54:04.089 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 771 k 2024-03-25T17:54:04.090 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.7 M 2024-03-25T17:54:04.091 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:54:08.544 INFO:teuthology.orchestra.run.smithi164.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 173 kB/s | 771 kB 00:04 2024-03-25T17:54:08.545 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:08.545 INFO:teuthology.orchestra.run.smithi164.stdout:Total 173 kB/s | 771 kB 00:04 2024-03-25T17:54:08.545 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:54:08.549 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:54:08.550 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:54:08.592 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:54:08.592 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:54:08.781 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:54:08.893 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:54:09.283 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:09.547 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:54:09.758 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:54:10.323 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:35:52 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:54:10.474 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:54:10.476 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:10.476 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 1.1 M 2024-03-25T17:54:10.476 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 4.3 M 2024-03-25T17:54:10.476 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:54:13.131 INFO:teuthology.orchestra.run.smithi164.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 434 kB/s | 1.1 MB 00:02 2024-03-25T17:54:13.131 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:13.132 INFO:teuthology.orchestra.run.smithi164.stdout:Total 433 kB/s | 1.1 MB 00:02 2024-03-25T17:54:13.132 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:54:13.136 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:54:13.136 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:54:13.179 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:54:13.179 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:54:13.401 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:54:13.505 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:13.926 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:14.191 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:14.191 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:14.191 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:54:14.191 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:14.191 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:14.192 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:54:14.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:54:14.955 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:35:56 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:54:15.106 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:54:15.106 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 337 k 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.0 M 2024-03-25T17:54:15.107 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:54:16.258 INFO:teuthology.orchestra.run.smithi164.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 293 kB/s | 337 kB 00:01 2024-03-25T17:54:16.259 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:16.259 INFO:teuthology.orchestra.run.smithi164.stdout:Total 293 kB/s | 337 kB 00:01 2024-03-25T17:54:16.260 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:54:16.263 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:54:16.264 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:54:16.302 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:54:16.303 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:54:16.444 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:54:16.549 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:16.920 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:17.178 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:54:17.374 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:54:17.938 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:35:59 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:54:18.088 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:54:18.089 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 509 k 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.5 M 2024-03-25T17:54:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:54:20.353 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 225 kB/s | 509 kB 00:02 2024-03-25T17:54:20.354 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:20.354 INFO:teuthology.orchestra.run.smithi164.stdout:Total 225 kB/s | 509 kB 00:02 2024-03-25T17:54:20.354 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:54:20.358 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:54:20.358 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:54:20.398 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:54:20.398 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:54:20.556 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:54:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:21.062 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:21.324 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:54:21.528 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:54:22.088 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:36:04 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:54:22.238 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 52 M 2024-03-25T17:54:22.239 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 298 M 2024-03-25T17:54:22.240 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:57:02.969 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 328 kB/s | 52 MB 02:40 2024-03-25T17:57:02.969 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:02.969 INFO:teuthology.orchestra.run.smithi164.stdout:Total 328 kB/s | 52 MB 02:40 2024-03-25T17:57:02.969 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:57:02.974 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:57:02.974 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:57:03.305 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:57:03.305 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:57:09.218 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:57:09.515 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:11.558 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:11.825 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:11.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:11.825 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:57:11.825 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:11.826 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:11.826 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:57:12.015 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:57:12.589 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 1:38:54 ago on Mon 25 Mar 2024 04:18:18 PM UTC. 2024-03-25T17:57:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 1.3 M 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 5.5 M 2024-03-25T17:57:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-25T17:57:18.935 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 207 kB/s | 1.3 MB 00:06 2024-03-25T17:57:18.936 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:18.936 INFO:teuthology.orchestra.run.smithi164.stdout:Total 207 kB/s | 1.3 MB 00:06 2024-03-25T17:57:18.937 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-25T17:57:18.940 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-25T17:57:18.941 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-25T17:57:18.986 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-25T17:57:18.986 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-25T17:57:19.239 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-25T17:57:19.352 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:19.781 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T17:57:20.051 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-25T17:57:20.230 DEBUG:teuthology.parallel:result is None 2024-03-25T17:58:00.489 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 358 kB/s | 287 MB 13:41 2024-03-25T17:58:00.490 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:00.490 INFO:teuthology.orchestra.run.smithi046.stdout:Total 358 kB/s | 287 MB 13:41 2024-03-25T17:58:00.490 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:58:00.492 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:58:00.492 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:58:02.284 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:58:02.284 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:58:33.381 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:58:33.478 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:40.866 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:41.143 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:58:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:41.143 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:58:41.360 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:58:41.937 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:39:59 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:58:42.091 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:58:42.092 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:42.092 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:58:42.092 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:42.092 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 5.9 M 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 28 M 2024-03-25T17:58:42.093 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:58:57.104 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 405 kB/s | 5.9 MB 00:15 2024-03-25T17:58:57.104 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:57.105 INFO:teuthology.orchestra.run.smithi046.stdout:Total 405 kB/s | 5.9 MB 00:15 2024-03-25T17:58:57.105 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:58:57.109 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:58:57.109 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:58:57.185 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:58:57.185 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:58:57.960 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:58:58.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:58.603 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:58.855 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:58.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:58.855 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:58:58.856 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:58.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:58.856 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:58:59.052 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:58:59.627 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:40:17 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:58:59.782 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:58:59.783 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:59.783 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 109 k 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 440 k 2024-03-25T17:58:59.784 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:59:00.623 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 130 kB/s | 109 kB 00:00 2024-03-25T17:59:00.623 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:00.624 INFO:teuthology.orchestra.run.smithi046.stdout:Total 129 kB/s | 109 kB 00:00 2024-03-25T17:59:00.624 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:59:00.628 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:59:00.628 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:59:00.667 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:59:00.668 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:59:00.787 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:59:00.899 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:01.414 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:01.671 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:01.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:01.672 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:59:01.672 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:01.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:01.672 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:59:01.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:59:02.432 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:40:20 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:59:02.588 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:02.589 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 5.4 M 2024-03-25T17:59:02.590 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 25 M 2024-03-25T17:59:02.590 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T17:59:06.434 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 5.4 MB 00:03 2024-03-25T17:59:06.435 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:06.435 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.4 MB/s | 5.4 MB 00:03 2024-03-25T17:59:06.436 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T17:59:06.440 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T17:59:06.440 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T17:59:06.512 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T17:59:06.512 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T17:59:07.214 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T17:59:07.322 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:07.875 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:08.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:08.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:08.142 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T17:59:08.142 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:08.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:08.142 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T17:59:08.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2-debuginfo 2024-03-25T17:59:08.919 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:40:26 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T17:59:09.074 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:59:09.075 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 44 M 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 253 M 2024-03-25T17:59:09.076 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:01:12.968 INFO:teuthology.orchestra.run.smithi046.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 366 kB/s | 44 MB 02:03 2024-03-25T18:01:12.969 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:12.969 INFO:teuthology.orchestra.run.smithi046.stdout:Total 366 kB/s | 44 MB 02:03 2024-03-25T18:01:12.969 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:01:12.972 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:01:12.972 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:01:13.284 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:01:13.285 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:01:18.611 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:01:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:20.551 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:01:20.830 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:01:21.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:01:21.600 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:42:39 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 52 M 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 308 M 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:24.037 INFO:teuthology.orchestra.run.smithi046.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 291 kB/s | 52 MB 03:02 2024-03-25T18:04:24.038 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:24.038 INFO:teuthology.orchestra.run.smithi046.stdout:Total 291 kB/s | 52 MB 03:02 2024-03-25T18:04:24.038 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:24.040 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:24.040 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:24.399 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:24.399 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:30.737 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:30.836 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:32.929 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:33.182 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:33.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:04:33.946 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:45:51 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:34.102 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 615 k 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.1 M 2024-03-25T18:04:34.103 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:36.296 INFO:teuthology.orchestra.run.smithi046.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 280 kB/s | 615 kB 00:02 2024-03-25T18:04:36.297 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:36.297 INFO:teuthology.orchestra.run.smithi046.stdout:Total 280 kB/s | 615 kB 00:02 2024-03-25T18:04:36.298 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:36.302 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:36.302 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:36.344 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:36.345 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:36.517 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:36.633 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:37.070 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:37.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:04:38.117 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:45:56 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:38.273 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:38.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:38.275 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 771 k 2024-03-25T18:04:38.275 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.7 M 2024-03-25T18:04:38.275 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:42.341 INFO:teuthology.orchestra.run.smithi046.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 190 kB/s | 771 kB 00:04 2024-03-25T18:04:42.342 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:42.342 INFO:teuthology.orchestra.run.smithi046.stdout:Total 190 kB/s | 771 kB 00:04 2024-03-25T18:04:42.342 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:42.346 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:42.346 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:42.390 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:42.390 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:42.580 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:42.743 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:43.133 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:43.388 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:43.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:04:44.170 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:46:02 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:44.326 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 1.1 M 2024-03-25T18:04:44.327 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 4.3 M 2024-03-25T18:04:44.328 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:47.936 INFO:teuthology.orchestra.run.smithi046.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 319 kB/s | 1.1 MB 00:03 2024-03-25T18:04:47.937 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:47.937 INFO:teuthology.orchestra.run.smithi046.stdout:Total 319 kB/s | 1.1 MB 00:03 2024-03-25T18:04:47.937 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:47.941 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:47.942 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:47.987 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:47.988 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:48.223 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:48.330 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:48.718 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:48.970 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:48.971 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:48.971 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:48.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:48.971 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:48.971 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:49.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:04:49.774 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:46:07 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:49.931 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:49.932 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 337 k 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.0 M 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:51.731 INFO:teuthology.orchestra.run.smithi046.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 188 kB/s | 337 kB 00:01 2024-03-25T18:04:51.731 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:51.732 INFO:teuthology.orchestra.run.smithi046.stdout:Total 187 kB/s | 337 kB 00:01 2024-03-25T18:04:51.732 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:51.736 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:51.736 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:51.777 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:51.777 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:51.920 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:52.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:52.416 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:52.671 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:52.671 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:52.672 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:52.672 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:52.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:52.672 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:52.901 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:04:53.475 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:46:11 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:53.630 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:53.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:53.632 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 509 k 2024-03-25T18:04:53.632 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 2.5 M 2024-03-25T18:04:53.632 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:04:56.028 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 212 kB/s | 509 kB 00:02 2024-03-25T18:04:56.028 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:56.029 INFO:teuthology.orchestra.run.smithi046.stdout:Total 212 kB/s | 509 kB 00:02 2024-03-25T18:04:56.029 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:04:56.033 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:04:56.033 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:04:56.074 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:04:56.074 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:04:56.241 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:04:56.354 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:56.765 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:57.029 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:57.029 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:57.029 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:04:57.029 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:57.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:57.030 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:04:57.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:04:57.814 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:46:15 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:04:57.969 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:04:57.970 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:04:57.971 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:04:57.971 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:04:57.971 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 52 M 2024-03-25T18:04:57.971 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 298 M 2024-03-25T18:04:57.971 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:07:30.417 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 346 kB/s | 52 MB 02:32 2024-03-25T18:07:30.417 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:30.417 INFO:teuthology.orchestra.run.smithi046.stdout:Total 346 kB/s | 52 MB 02:32 2024-03-25T18:07:30.417 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:07:30.420 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:07:30.420 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:07:30.772 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:07:30.772 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:07:37.024 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:07:37.116 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:39.173 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:39.433 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:39.434 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:39.434 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:07:39.434 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:39.434 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:39.434 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:07:39.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:07:40.227 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 1:48:58 ago on Mon 25 Mar 2024 04:18:42 PM UTC. 2024-03-25T18:07:40.383 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-25T18:07:40.384 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:07:40.384 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-03-25T18:07:40.384 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:07:40.384 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 1.3 M 2024-03-25T18:07:40.385 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 5.5 M 2024-03-25T18:07:40.386 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-25T18:07:48.617 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 156 kB/s | 1.3 MB 00:08 2024-03-25T18:07:48.618 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:48.618 INFO:teuthology.orchestra.run.smithi046.stdout:Total 156 kB/s | 1.3 MB 00:08 2024-03-25T18:07:48.619 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-25T18:07:48.623 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-25T18:07:48.623 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-25T18:07:48.669 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-25T18:07:48.669 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-25T18:07:48.918 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-25T18:07:49.019 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:49.438 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-25T18:07:49.927 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:49.928 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:07:50.145 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:07:50.166 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:50.166 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:50.166 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:07:50.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:07:50.309 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:07:50.334 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:50.334 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:07:50.334 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:07:50.336 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:07:50.336 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:50.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:07:50.363 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:50.364 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:07:50.405 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:07:50.406 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:50.406 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:07:50.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:07:50.501 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:50.501 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:07:50.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:07:50.602 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:07:50.602 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:50.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:07:50.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:07:50.696 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:50.696 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:07:50.725 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:07:50.794 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:07:50.794 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:50.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:07:50.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:07:50.889 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:50.889 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:07:50.917 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:07:50.986 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:07:51.071 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:07:51.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-03-25T18:07:51.074 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-03-25T18:07:51.102 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:07:51.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:07:51.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:07:51.169 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:07:51.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:07:51.208 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:07:51.267 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:07:51.267 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:07:51.267 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621253', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGgMmjL9vIjb25dSI+q3qzntVgzJ5w3kec8X6xKB/SOOyIFMDrMxKtF8cO1JbTgGDF3xdWS63ZPWI7+5Td7VeE=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsZiptMAL9EmMGCc4pX0U3Pjy3haBJUASvF1lidQ3R5LIIt+aMAsppp9t5Nrftpp9rpOejdZcHctBsAY5nXXc4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:07:51.268 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:07:51.276 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:07:51.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:07:51.330 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:07:51.357 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:51.357 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:07:51.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:07:51.409 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-03-25T18:07:51.463 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:07:51.463 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:51.463 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 811 Links: 1 2024-03-25T18:07:51.463 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:51.464 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:51.464 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-25 16:24:37.055204538 +0000 2024-03-25T18:07:51.464 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-25 16:17:34.921606755 +0000 2024-03-25T18:07:51.464 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-25 16:17:34.921606755 +0000 2024-03-25T18:07:51.464 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-25 16:17:34.921606755 +0000 2024-03-25T18:07:51.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:07:51.528 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-25T18:07:51.528 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-25T18:07:51.528 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 7.9096e-05 s, 6.5 MB/s 2024-03-25T18:07:51.529 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:07:51.585 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-03-25T18:07:51.639 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:07:51.639 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 832 Links: 1 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-25 16:24:37.056204527 +0000 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-25 16:17:35.266603095 +0000 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-25 16:17:35.266603095 +0000 2024-03-25T18:07:51.640 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-25 16:17:35.266603095 +0000 2024-03-25T18:07:51.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:07:51.705 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-25T18:07:51.705 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-25T18:07:51.705 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 7.967e-05 s, 6.4 MB/s 2024-03-25T18:07:51.706 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:07:51.761 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 849 Links: 1 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-25 16:24:37.056204527 +0000 2024-03-25T18:07:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-25 16:17:35.593599625 +0000 2024-03-25T18:07:51.816 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-25 16:17:35.593599625 +0000 2024-03-25T18:07:51.816 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-25 16:17:35.593599625 +0000 2024-03-25T18:07:51.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:07:51.879 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-25T18:07:51.880 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-25T18:07:51.880 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.8708e-05 s, 5.8 MB/s 2024-03-25T18:07:51.881 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:07:51.936 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-03-25T18:07:51.990 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:07:51.990 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-25 16:24:37.057204516 +0000 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-25 16:17:35.925596103 +0000 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-25 16:17:35.925596103 +0000 2024-03-25T18:07:51.991 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-25 16:17:35.925596103 +0000 2024-03-25T18:07:51.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:07:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-25T18:07:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-25T18:07:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.6259e-05 s, 5.3 MB/s 2024-03-25T18:07:52.057 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:07:52.112 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:52.112 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:52.112 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:07:52.130 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:07:52.130 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-03-25T18:07:52.186 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:07:52.186 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:52.186 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-25 16:21:31.565155782 +0000 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-25 16:17:25.597925432 +0000 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-25 16:17:25.597925432 +0000 2024-03-25T18:07:52.187 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-25 16:17:25.597925432 +0000 2024-03-25T18:07:52.187 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:07:52.254 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-25T18:07:52.254 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-25T18:07:52.255 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000219997 s, 2.3 MB/s 2024-03-25T18:07:52.256 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:07:52.312 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-03-25T18:07:52.367 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-25 16:21:31.565155782 +0000 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-25 16:17:25.952919871 +0000 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-25 16:17:25.952919871 +0000 2024-03-25T18:07:52.368 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-25 16:17:25.952919871 +0000 2024-03-25T18:07:52.368 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:07:52.434 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-25T18:07:52.434 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-25T18:07:52.434 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000210447 s, 2.4 MB/s 2024-03-25T18:07:52.435 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:07:52.492 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-03-25T18:07:52.546 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-25 16:21:31.566155767 +0000 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-25 16:17:26.305914342 +0000 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-25 16:17:26.305914342 +0000 2024-03-25T18:07:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-25 16:17:26.305914342 +0000 2024-03-25T18:07:52.547 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:07:52.612 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-25T18:07:52.612 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-25T18:07:52.613 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000207096 s, 2.5 MB/s 2024-03-25T18:07:52.614 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:07:52.669 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-03-25T18:07:52.725 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:07:52.725 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:07:52.725 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:07:52.725 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:07:52.726 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:07:52.726 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-25 16:21:31.566155767 +0000 2024-03-25T18:07:52.726 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-25 16:17:26.638909126 +0000 2024-03-25T18:07:52.726 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-25 16:17:26.638909126 +0000 2024-03-25T18:07:52.726 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-25 16:17:26.638909126 +0000 2024-03-25T18:07:52.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:07:52.790 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-25T18:07:52.790 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-25T18:07:52.790 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000205646 s, 2.5 MB/s 2024-03-25T18:07:52.791 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:07:52.847 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:52.847 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:07:52.847 INFO:tasks.ceph:Generating config... 2024-03-25T18:07:52.849 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:07:52.849 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:07:52.849 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:07:52.849 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:07:52.849 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:07:52.850 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:07:52.850 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:07:52.851 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:07:52.852 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:07:52.852 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:07:52.853 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:52.914 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:07:52.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:07:52.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:52.988 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.46:6789]'), ('mon.c', '[v1:172.21.15.46:6790]'), ('mon.b', '[v1:172.21.15.164:6789]')] 2024-03-25T18:07:52.988 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.46:6789],[v1:172.21.15.46:6790],[v1:172.21.15.164:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:07:52.989 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:52.989 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:07:53.043 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.46:6789]' --addv c '[v1:172.21.15.46:6790]' --addv b '[v1:172.21.15.164:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.127 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.127 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 17 (quincy) 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 3 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:07:53.128 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:07:53.129 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:07:53.143 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a6bb6b92-3bc5-4b76-b609-dd01555b77cc... 2024-03-25T18:07:53.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:07:53.185 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-03-25T18:07:53.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-03-25T18:07:53.234 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "[v1:172.21.15.46:6789],[v1:172.21.15.46:6790],[v1:172.21.15.164:6789]" 2024-03-25T18:07:53.235 INFO:teuthology.orchestra.run.smithi046.stdout: mon client directed command retry = 5 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: mon election default strategy = 3 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: ms bind msgr2 = False 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject socket failures = 5000 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: ms type = async 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd heartbeat grace = 80 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-03-25T18:07:53.236 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-03-25T18:07:53.237 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = stupid 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore zero block detection = True 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 1 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 1 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 20 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-03-25T18:07:53.238 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd class default list = * 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd class load list = * 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd client watch timeout = 120 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify cached snaps = True 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify missing on start = True 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd fast shutdown = False 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd max markdown count = 1000 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock override recovery settings = True 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue = debug_random 2024-03-25T18:07:53.239 INFO:teuthology.orchestra.run.smithi046.stdout: osd op queue cut off = debug_random 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: osd shutdown pgref assert = True 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-03-25T18:07:53.240 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-03-25T18:07:53.241 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon min osdmap epochs = 50 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd crush smoke test = False 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune interval = 2 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune min = 15 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon scrub interval = 300 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on pool no app = False 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: paxos service trim min = 10 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-03-25T18:07:53.242 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 20 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout: debug objecter = 20 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout: debug rados = 20 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:07:53.243 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:07:53.244 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:07:53.245 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "[v1:172.21.15.46:6789],[v1:172.21.15.46:6790],[v1:172.21.15.164:6789]" 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: mon election default strategy = 3 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: ms bind msgr2 = False 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 5000 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: ms type = async 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: osd heartbeat grace = 80 2024-03-25T18:07:53.246 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-25T18:07:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = stupid 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore zero block detection = True 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 10 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: osd class default list = * 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: osd class load list = * 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: osd client watch timeout = 120 2024-03-25T18:07:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify cached snaps = True 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify missing on start = True 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd fast shutdown = False 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd max markdown count = 1000 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock override recovery settings = True 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue = debug_random 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue cut off = debug_random 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: osd shutdown pgref assert = True 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-25T18:07:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon min osdmap epochs = 50 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd crush smoke test = False 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune interval = 2 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune min = 15 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:07:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: mon scrub interval = 300 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: paxos service trim min = 10 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: debug objecter = 20 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: debug rados = 20 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-25T18:07:53.253 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-25T18:07:53.255 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:07:53.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:07:53.309 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:07:53.309 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:53.309 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:07:53.323 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:53.324 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:07:53.378 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:07:53.378 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:53.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:07:53.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:53.457 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:07:53.457 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.512 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:07:53.512 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:53.512 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:07:53.512 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:07:53.552 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:07:53.552 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:53.607 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:07:53.608 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:07:53.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:07:53.662 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:07:53.664 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:07:53.723 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:07:53.726 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:07:53.726 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:07:53.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:07:53.768 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:07:53.783 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:07:53.824 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:07:53.839 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:07:53.840 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:07:53.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:53.866 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:53.866 INFO:tasks.ceph:role: osd.0 2024-03-25T18:07:53.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:07:53.866 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:53.932 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:53.933 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:53.964 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-03-25T18:07:53.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-03-25T18:07:53.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:54.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:54.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:54.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:54.174 INFO:tasks.ceph:role: osd.1 2024-03-25T18:07:54.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:07:54.174 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:07:54.240 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:54.240 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:54.240 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:54.241 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:54.273 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-03-25T18:07:54.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-03-25T18:07:54.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:54.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:54.415 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:54.482 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:54.482 INFO:tasks.ceph:role: osd.2 2024-03-25T18:07:54.482 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:07:54.482 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:54.548 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:54.549 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:54.549 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:54.580 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-03-25T18:07:54.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-03-25T18:07:54.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:54.653 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:54.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:54.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:54.792 INFO:tasks.ceph:role: osd.3 2024-03-25T18:07:54.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:07:54.792 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:54.858 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:54.859 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:54.859 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:54.890 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-03-25T18:07:54.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-03-25T18:07:54.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:54.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:55.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.124+0000 7f26af99e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:07:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.125+0000 7f26af99e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:07:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.125+0000 7f26af99e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.125+0000 7f26af99e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.125+0000 7f26af99e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:07:55.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:55.125+0000 7f26af99e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:07:57.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:07:57.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:58.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.088+0000 7f2efdd19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:07:58.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.089+0000 7f2efdd19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:07:58.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.089+0000 7f2efdd19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:58.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.089+0000 7f2efdd19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:58.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.089+0000 7f2efdd19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:07:58.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:07:58.089+0000 7f2efdd19640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:08:00.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:08:00.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.047+0000 7ff1d3cc2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.047+0000 7ff1d3cc2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.047+0000 7ff1d3cc2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.047+0000 7ff1d3cc2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.047+0000 7ff1d3cc2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:01.048+0000 7ff1d3cc2640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:08:03.884 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:08:03.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.007+0000 7f8afeea2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:08:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.008+0000 7f8afeea2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:08:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.008+0000 7f8afeea2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.008+0000 7f8afeea2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.008+0000 7f8afeea2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:04.008+0000 7f8afeea2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:08:06.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:08:06.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:08:06.914 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:06.914 INFO:tasks.ceph:role: osd.4 2024-03-25T18:08:06.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:08:06.914 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:06.991 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:06.992 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:06.992 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:06.992 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:06.992 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:07.506 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-25T18:08:07.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-25T18:08:07.508 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:08:07.665 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:08:07.704 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:08:07.775 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:07.775 INFO:tasks.ceph:role: osd.5 2024-03-25T18:08:07.775 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:08:07.776 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:07.852 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:07.853 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:07.853 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:08.294 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-25T18:08:08.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-25T18:08:08.296 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:08:08.395 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:08:08.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:08:08.493 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:08.493 INFO:tasks.ceph:role: osd.6 2024-03-25T18:08:08.493 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:08:08.493 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:08:08.567 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:08.567 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:08.567 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:08.567 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:08.568 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:09.061 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-25T18:08:09.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-25T18:08:09.063 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:08:09.208 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:08:09.241 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:08:09.308 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:09.308 INFO:tasks.ceph:role: osd.7 2024-03-25T18:08:09.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:08:09.309 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:09.383 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:09.384 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:09.384 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:09.906 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-25T18:08:09.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-25T18:08:09.908 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:08:10.057 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:08:10.092 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:10.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.201+0000 7fcc10499640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:08:10.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.201+0000 7fcc10499640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:08:10.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.201+0000 7fcc10499640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:08:10.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.201+0000 7fcc10499640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:08:10.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.201+0000 7fcc10499640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:08:10.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:10.202+0000 7fcc10499640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:08:13.049 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:08:13.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:13.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.174+0000 7f4f7982a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:08:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.174+0000 7f4f7982a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:08:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.174+0000 7f4f7982a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:08:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.174+0000 7f4f7982a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:08:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.174+0000 7f4f7982a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:08:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:13.175+0000 7f4f7982a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:08:16.015 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:08:16.047 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:16.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.139+0000 7fbed7024640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:08:16.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.139+0000 7fbed7024640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:08:16.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.139+0000 7fbed7024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:08:16.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.139+0000 7fbed7024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:08:16.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.139+0000 7fbed7024640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:08:16.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:16.140+0000 7fbed7024640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:08:18.989 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:08:19.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:19.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:08:19.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:08:19.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:08:19.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:08:19.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:08:19.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:19.108+0000 7efd6e068640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:08:21.947 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:08:21.978 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:08:21.978 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:21.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:08:22.011 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:22.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:08:22.081 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:22.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:08:22.151 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:22.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:08:22.220 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:22.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:08:22.289 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.289 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:08:22.320 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.321 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:08:22.392 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.393 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:08:22.464 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.464 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:08:22.533 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:08:22.604 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:22.604 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:08:22.621 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:22.621 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:08:22.660 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:08:22.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:08:22.664 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB5vQFmiCZ5JxAAS4AEUjMzYOztub9HRKObIA== 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB7vQFmGX2KBxAALrqlkt9YrZl2JIRT+0JimQ== 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB+vQFm/V5XBRAAUTQdwe6oSWHALjVxms9L0w== 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCBvQFmcAffAhAAwehWwBwDw7iseLG3Dl5oFQ== 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCEvQFmlAKDABAAk5ZI1fMPbdux2EkCzrkEgg== 2024-03-25T18:08:22.703 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB5vQFmrToqKxAAn7dHwBVXiWWhLe/nKLH+RQ== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCKvQFm78MIDBAA7pg+3qISQrmZpt8dF5fVYA== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCNvQFmUdtuChAAn04TWYc4C73wM0TkCPXWdg== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCQvQFmoMVQCBAArNFPFG9YD/GIADQ5dE2xyw== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCTvQFmwex5BhAAlnJyUVdltt80ySuI4NRtUw== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB5vQFmG5HPLRAAVVlRmTmZJsbFEw9vD0+ywA== 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:[client.1] 2024-03-25T18:08:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB5vQFmymExMRAAKC2urfiwdl2bf8KyWkSNPw== 2024-03-25T18:08:22.727 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.y] 2024-03-25T18:08:22.727 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB5vQFmiCZ5JxAAS4AEUjMzYOztub9HRKObIA== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB7vQFmGX2KBxAALrqlkt9YrZl2JIRT+0JimQ== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB+vQFm/V5XBRAAUTQdwe6oSWHALjVxms9L0w== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCBvQFmcAffAhAAwehWwBwDw7iseLG3Dl5oFQ== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCEvQFmlAKDABAAk5ZI1fMPbdux2EkCzrkEgg== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB5vQFmrToqKxAAn7dHwBVXiWWhLe/nKLH+RQ== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCKvQFm78MIDBAA7pg+3qISQrmZpt8dF5fVYA== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCNvQFmUdtuChAAn04TWYc4C73wM0TkCPXWdg== 2024-03-25T18:08:22.728 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.6] 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCQvQFmoMVQCBAArNFPFG9YD/GIADQ5dE2xyw== 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.7] 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCTvQFmwex5BhAAlnJyUVdltt80ySuI4NRtUw== 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB5vQFmG5HPLRAAVVlRmTmZJsbFEw9vD0+ywA== 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2024-03-25T18:08:22.729 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB5vQFmymExMRAAKC2urfiwdl2bf8KyWkSNPw== 2024-03-25T18:08:22.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:08:22.748 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:08:22.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.828 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.883 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.937 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:22.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:08:23.044 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:08:23.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.098 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.151 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.259 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:08:23.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:08:23.312 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:08:23.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:08:23.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:08:23.423 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:08:23.424 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:08:23.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:08:23.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:08:23.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:08:23.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:08:23.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:08:23.759 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:08:23.786 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:08:23.883 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:08:23.910 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:23.912 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:23.964 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:08:23.964 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:08:23.964 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:08:23.965 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:08:23.967 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:08:23.967 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:08:23.967 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:08:23.967 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:08:23.969 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:08:23.969 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:08:23.970 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:08:23.970 DEBUG:teuthology.orchestra.run.smithi164:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:08:24.006 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:08:24.006 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:08:24.006 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:08:24.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:08:24.009 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:08:24.009 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:08:24.009 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:08:24.011 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:08:24.011 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:08:24.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:08:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 10 monclient: get_monmap_and_config 2024-03-25T18:08:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 10 monclient: build_initial_monmap 2024-03-25T18:08:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 10 monclient: monmap: 2024-03-25T18:08:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:08:24.113950+0000 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:08:24.113950+0000 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:24.114 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 1 Processor -- start 2024-03-25T18:08:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 1 -- start start 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.113+0000 7f0be57e6640 10 monclient: init 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-b con 0x7f0be00b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-a con 0x7f0be00a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00a9d60 con 0x7f0be00a96b0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be013ff10 con 0x7f0be0142a20 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00aa1f0 con 0x7f0be00b4700 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0be57e6640 10 monclient(hunting): authenticate will time out at 7213.376071s 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0be0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0be013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:24.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.114+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:24.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.315+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0be0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:24.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.315+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0be013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:24.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.315+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:24.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.716+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0be0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:24.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.716+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0be013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:24.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:24.716+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:25.517+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0be0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:25.517+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0be013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:25.517+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): tick 2024-03-25T18:08:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0be0142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0be013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0be00a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0be00b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b4ae0 con 0x7f0be00a96b0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00a9d60 con 0x7f0be0142a20 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be013ff10 con 0x7f0be00b4700 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdcff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bde7fc640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.114+0000 7f0bdeffd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:27.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.315+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:27.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.315+0000 7f0bde7fc640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:27.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.315+0000 7f0bdeffd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.716+0000 7f0bde7fc640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.716+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:27.716+0000 7f0bdeffd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:28.517+0000 7f0bde7fc640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:28.517+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:28.517+0000 7f0bdeffd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): tick 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0be00a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0be00b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00aa1f0 con 0x7f0be00a96b0 2024-03-25T18:08:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b4ae0 con 0x7f0be0142a20 2024-03-25T18:08:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00a9d60 con 0x7f0be00b4700 2024-03-25T18:08:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdcff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.114+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.315+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:30.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.315+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:30.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.315+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:30.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.716+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:30.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.716+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:30.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:30.716+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:31.017 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:00:06.888 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:31.327 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:00:07.199 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:31.480 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:00:07.398 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:31.481 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:00:07.400 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:31.517+0000 7f0bdeffd640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6789/0 2024-03-25T18:08:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:31.517+0000 7f0bdf7fe640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.46:6790/0 2024-03-25T18:08:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:31.517+0000 7f0bde7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-03-25T18:08:31.808 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:00:07.725 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:31.808 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:00:07.726 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 10 monclient(hunting): tick 2024-03-25T18:08:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.114+0000 7f0bdcff9640 1 -- >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0be00b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0be00a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be013ff10 con 0x7f0be00a96b0 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00aa1f0 con 0x7f0be0142a20 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b4ae0 con 0x7f0be00b4700 2024-03-25T18:08:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.115+0000 7f0bdcff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:33.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.163+0000 7f0bdeffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 0x7f0bcc010790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:51292/0 (socket says 172.21.15.46:51292) 2024-03-25T18:08:33.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.163+0000 7f0bdeffd640 1 -- 172.21.15.46:0/2727613466 learned_addr learned my addr 172.21.15.46:0/2727613466 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.531+0000 7f0bdf7fe640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:08:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.531+0000 7f0bdf7fe640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:08:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.531+0000 7f0bdf7fe640 1 --1- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 0x7f0bcc00ff70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:08:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.552+0000 7f0bde7fc640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:08:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.552+0000 7f0bde7fc640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:08:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:33.552+0000 7f0bde7fc640 1 --1- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 0x7f0bcc00c770 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): tick 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0be00b4700 legacy=0x7f0bcc00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc010790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0be00a96b0 legacy=0x7f0bcc00ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc009f10 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc0120c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00a9d60 con 0x7f0bcc0120c0 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc00a470 con 0x7f0be0142a20 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc00f010 con 0x7f0bcc009f10 2024-03-25T18:08:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:36.115+0000 7f0bdcff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): tick 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0bcc009f10 legacy=0x7f0bcc0066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc010790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0bcc0120c0 legacy=0x7f0bcc00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc009f10 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc00f1a0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0be0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc010270 con 0x7f0bcc00f1a0 2024-03-25T18:08:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc00fc90 con 0x7f0be0142a20 2024-03-25T18:08:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc00fe70 con 0x7f0bcc009f10 2024-03-25T18:08:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:39.115+0000 7f0bdcff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871570706 0 0) 0x7f0bcc00fe70 con 0x7f0bcc009f10 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:08:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.968+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc8003520 con 0x7f0bcc009f10 2024-03-25T18:08:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436299834 0 0) 0x7f0bcc00fc90 con 0x7f0be0142a20 2024-03-25T18:08:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:41.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:08:41.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:41.990+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc00fe70 con 0x7f0be0142a20 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504133349 0 0) 0x7f0bcc010270 con 0x7f0bcc00f1a0 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:08:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.002+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc00fc90 con 0x7f0bcc00f1a0 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520989717 0 0) 0x7f0bc8003520 con 0x7f0bcc009f10 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.164:6789/0 conn(0x7f0be0142a20 legacy=0x7f0bcc010790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6789/0 conn(0x7f0bcc00f1a0 legacy=0x7f0bcc00f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 monclient: found mon.noname-b 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc00d780 con 0x7f0bcc009f10 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:42.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0bd0004150 con 0x7f0bcc009f10 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0be57e6640 5 monclient: authenticate success, global_id 4103 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0be57e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0bddffb640 10 monclient: dump: 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:42.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.017+0000 7f0be57e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bd0004320 con 0x7f0bcc009f10 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 1 -- 172.21.15.46:0/2727613466 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0bd0004730 con 0x7f0bcc009f10 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0bddffb640 10 monclient: dump: 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0be57e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:42.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.053+0000 7f0be57e6640 4 set_mon_vals no callback set 2024-03-25T18:08:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 10 monclient: shutdown 2024-03-25T18:08:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 1 -- 172.21.15.46:0/2727613466 >> v1:172.21.15.46:6790/0 conn(0x7f0bcc009f10 legacy=0x7f0bcc0066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 1 -- 172.21.15.46:0/2727613466 shutdown_connections 2024-03-25T18:08:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 1 -- 172.21.15.46:0/2727613466 >> 172.21.15.46:0/2727613466 conn(0x7f0be00668e0 msgr2=0x7f0be0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 1 -- 172.21.15.46:0/2727613466 shutdown_connections 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.054+0000 7f0be57e6640 1 -- 172.21.15.46:0/2727613466 wait complete. 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.055+0000 7f0be57e6640 10 monclient: build_initial_monmap 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.055+0000 7f0be57e6640 10 monclient: monmap: 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:42.058 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.055+0000 7f0be57e6640 1 librados: starting msgr at 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.055+0000 7f0be57e6640 1 librados: starting objecter 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.055+0000 7f0be57e6640 1 Processor -- start 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 -- start start 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 librados: setting wanted keys 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 librados: calling monclient init 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient: init 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-c con 0x7f0bcc009f10 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc00f1a0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient(hunting): picked mon.noname-a con 0x7f0be0142a20 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b0ff0 con 0x7f0be0142a20 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b1290 con 0x7f0bcc00f1a0 2024-03-25T18:08:42.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b1420 con 0x7f0bcc009f10 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.056+0000 7f0be57e6640 10 monclient(hunting): authenticate will time out at 7231.318146s 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bdf7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0be0142a20 0x7f0be01d83c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48430/0 (socket says 172.21.15.46:48430) 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bdf7fe640 1 -- 172.21.15.46:0/1145951707 learned_addr learned my addr 172.21.15.46:0/1145951707 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bc77fe640 10 client.?.objecter ms_handle_connect 0x7f0be0142a20 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bc77fe640 10 client.?.objecter ms_handle_connect 0x7f0bcc00f1a0 2024-03-25T18:08:42.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.057+0000 7f0bc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:42.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.108+0000 7f0bc77fe640 10 client.?.objecter ms_handle_connect 0x7f0bcc009f10 2024-03-25T18:08:42.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.108+0000 7f0bc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309224553 0 0) 0x7f0be00b0ff0 con 0x7f0be0142a20 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:08:42.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.110+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc010270 con 0x7f0be0142a20 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372800593 0 0) 0x7f0be00b1420 con 0x7f0bcc009f10 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:08:42.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.114+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be00b0ff0 con 0x7f0bcc009f10 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505884219 0 0) 0x7f0be00b1290 con 0x7f0bcc00f1a0 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:08:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.130+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be00b1420 con 0x7f0bcc00f1a0 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 916506682 0 0) 0x7f0be00b0ff0 con 0x7f0bcc009f10 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0be00b1290 con 0x7f0bcc009f10 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0bd0004630 con 0x7f0bcc009f10 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 10 monclient(hunting): dump: 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:42.142 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:42.142 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:42.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.140+0000 7f0bc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 843359189 0 0) 0x7f0be00b1290 con 0x7f0bcc009f10 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 >> v1:172.21.15.164:6789/0 conn(0x7f0bcc00f1a0 legacy=0x7f0be01d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 >> v1:172.21.15.46:6789/0 conn(0x7f0be0142a20 legacy=0x7f0be01d83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 1 monclient: found mon.c 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00b15b0 con 0x7f0bcc009f10 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0bc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 5 monclient: authenticate success, global_id 4109 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 10 monclient: _renew_subs 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be01c8690 con 0x7f0bcc009f10 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 10 monclient: _renew_subs 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be00b0960 con 0x7f0bcc009f10 2024-03-25T18:08:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.265+0000 7f0be57e6640 1 librados: init done 2024-03-25T18:08:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.266+0000 7f0be57e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.266+0000 7f0be57e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.266+0000 7f0be57e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.266+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be004f730 con 0x7f0bcc009f10 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.449+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bd0004a40 con 0x7f0bcc009f10 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.449+0000 7f0bc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.449+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0bd0004d50 con 0x7f0bcc009f10 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.449+0000 7f0bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 10 monclient: dump: 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0bd0005240 con 0x7f0bcc009f10 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2553656591 0 0) 0x7f0bd00065f0 con 0x7f0bcc009f10 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 10 client.4109.objecter ms_dispatch 0x7f0be0000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bdd7fa640 4 set_mon_vals no callback set 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 3 client.4109.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 3 client.4109.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0bd00069d0 con 0x7f0bcc009f10 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.450+0000 7f0bc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.522+0000 7f0be57e6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.522+0000 7f0be57e6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.522+0000 7f0be57e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.522+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0be00b17c0 con 0x7f0bcc009f10 2024-03-25T18:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.724+0000 7f0bc77fe640 1 -- 172.21.15.46:0/1145951707 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0bd0005760 con 0x7f0bcc009f10 2024-03-25T18:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.724+0000 7f0bc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.724+0000 7f0bc77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-03-25T18:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 10 librados: watch_flush enter 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 10 librados: watch_flush exit 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 10 client.4109.objecter successfully canceled tick 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 10 monclient: shutdown 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 >> v1:172.21.15.46:6790/0 conn(0x7f0bcc009f10 legacy=0x7f0be013a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 shutdown_connections 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.725+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 >> 172.21.15.46:0/1145951707 conn(0x7f0be00668e0 msgr2=0x7f0be00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.726+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 shutdown_connections 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.726+0000 7f0be57e6640 1 -- 172.21.15.46:0/1145951707 wait complete. 2024-03-25T18:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:42.726+0000 7f0be57e6640 1 librados: shutdown 2024-03-25T18:08:42.737 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:08:42.737 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:08:42.737 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:42.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:08:42.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:42.748+0000 7f9d64858040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:08:42.765 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:42.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:08:42.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:42.805+0000 7f9d64858040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:08:42.818 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:42.818+0000 7f887063e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:08:42.833 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:42.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:08:42.880 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:42.880+0000 7f887063e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:08:42.900 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-25T18:08:42.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:08:42.969 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:42.969 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:08:42.997 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:42.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:08:43.064 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:43.064 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:08:43.130 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-25T18:08:43.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:08:43.198 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 5134f463-b5c2-47f0-9dff-01724449ce31 0 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.336+0000 7f25e90b3640 10 monclient: get_monmap_and_config 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.336+0000 7f25e90b3640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.336+0000 7f25e90b3640 10 monclient: monmap: 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:43.338166+0000 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:43.338166+0000 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:43.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 1 Processor -- start 2024-03-25T18:08:43.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 1 -- start start 2024-03-25T18:08:43.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient: init 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-b con 0x7f25e40b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f25e4142a20 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f25e40a96b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40a9d60 con 0x7f25e4142a20 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e413ff10 con 0x7f25e40a96b0 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40aa1f0 con 0x7f25e40b4700 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.337+0000 7f25e90b3640 10 monclient(hunting): authenticate will time out at 7246.633517s 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.338+0000 7f25e359e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f25e40a96b0 0x7f25e413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59282/0 (socket says 172.21.15.164:59282) 2024-03-25T18:08:43.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.338+0000 7f25e359e640 1 -- 172.21.15.164:0/3115793985 learned_addr learned my addr 172.21.15.164:0/3115793985 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:43.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111486034 0 0) 0x7f25e413ff10 con 0x7f25e40a96b0 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc003520 con 0x7f25e40a96b0 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080894028 0 0) 0x7f25e40a9d60 con 0x7f25e4142a20 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e413ff10 con 0x7f25e4142a20 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017683920 0 0) 0x7f25e40aa1f0 con 0x7f25e40b4700 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e40a9d60 con 0x7f25e40b4700 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e259c640 1 -- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 legacy=0x7f25e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e259c640 1 -- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 legacy=0x7f25e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e259c640 1 --1- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 0x7f25e413f0f0 :-1 s=OPENED pgs=24 cs=1 l=1).handle_message read tag failed 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e259c640 1 --1- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 0x7f25e413f0f0 :-1 s=OPENED pgs=24 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:08:43.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.341+0000 7f25e1d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.46:6789/0 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368021016 0 0) 0x7f25e40a9d60 con 0x7f25e40b4700 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 >> v1:172.21.15.164:6789/0 conn(0x7f25e40a96b0 legacy=0x7f25e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 legacy=0x7f25e413f0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40b4ae0 con 0x7f25e40b4700 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f25d40015b0 con 0x7f25e40b4700 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e90b3640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e90b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.344+0000 7f25e1d9b640 10 monclient: dump: 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.347 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.345+0000 7f25e90b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d4001710 con 0x7f25e40b4700 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 1 -- 172.21.15.164:0/3115793985 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f25d4001b60 con 0x7f25e40b4700 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e1d9b640 10 monclient: dump: 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.349 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e90b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.346+0000 7f25e90b3640 4 set_mon_vals no callback set 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 10 monclient: shutdown 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 1 -- 172.21.15.164:0/3115793985 >> v1:172.21.15.46:6790/0 conn(0x7f25e40b4700 legacy=0x7f25e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 1 -- 172.21.15.164:0/3115793985 shutdown_connections 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 1 -- 172.21.15.164:0/3115793985 >> 172.21.15.164:0/3115793985 conn(0x7f25e40668e0 msgr2=0x7f25e4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 1 -- 172.21.15.164:0/3115793985 shutdown_connections 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.347+0000 7f25e90b3640 1 -- 172.21.15.164:0/3115793985 wait complete. 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 10 monclient: monmap: 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 librados: starting msgr at 2024-03-25T18:08:43.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 librados: starting objecter 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 Processor -- start 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 -- start start 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 librados: setting wanted keys 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 1 librados: calling monclient init 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.348+0000 7f25e90b3640 10 monclient: init 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-b con 0x7f25e40a96b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f25e40b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f25e4142a20 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e41375f0 con 0x7f25e4142a20 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e412dcf0 con 0x7f25e40a96b0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e41361e0 con 0x7f25e40b4700 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e90b3640 10 monclient(hunting): authenticate will time out at 7246.645067s 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e259c640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f25e40b4700 0x7f25e412ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.164:39966/0 (socket says 172.21.15.164:39966) 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e2d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f25e40a96b0 0x7f25e4136ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59292/0 (socket says 172.21.15.164:59292) 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.349+0000 7f25e259c640 1 -- 172.21.15.164:0/286989962 learned_addr learned my addr 172.21.15.164:0/286989962 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.350+0000 7f25cb7fe640 10 client.?.objecter ms_handle_connect 0x7f25e40a96b0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.350+0000 7f25cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.350+0000 7f25cb7fe640 10 client.?.objecter ms_handle_connect 0x7f25e40b4700 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.350+0000 7f25cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445534837 0 0) 0x7f25e41361e0 con 0x7f25e40b4700 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0004eb0 con 0x7f25e40b4700 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922334487 0 0) 0x7f25e412dcf0 con 0x7f25e40a96b0 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:08:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.352+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e41361e0 con 0x7f25e40a96b0 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540084611 0 0) 0x7f25d0004eb0 con 0x7f25e40b4700 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e412dcf0 con 0x7f25e40b4700 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f25d8001af0 con 0x7f25e40b4700 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.358 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.357+0000 7f25cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2870231847 0 0) 0x7f25e41361e0 con 0x7f25e40a96b0 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25d0004eb0 con 0x7f25e40a96b0 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f25d4001a20 con 0x7f25e40a96b0 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:08:43.360 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.361 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881564209 0 0) 0x7f25e412dcf0 con 0x7f25e40b4700 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 >> v1:172.21.15.164:6789/0 conn(0x7f25e40a96b0 legacy=0x7f25e4136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 >> v1:172.21.15.46:6789/0 conn(0x7f25e4142a20 legacy=0x7f25e412d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 1 monclient: found mon.c 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.359+0000 7f25cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e41364c0 con 0x7f25e40b4700 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _renew_subs 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e4134d80 con 0x7f25e40b4700 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _renew_subs 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e4142090 con 0x7f25e40b4700 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 1 librados: init done 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.360+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e4134fc0 con 0x7f25e40b4700 2024-03-25T18:08:43.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.383+0000 7f9d64858040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:08:43.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.436+0000 7f9d64858040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:08:43.456 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.455+0000 7f887063e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d8001d70 con 0x7f25e40b4700 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f25d80038c0 con 0x7f25e40b4700 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 10 monclient: dump: 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f25d8003e70 con 0x7f25e40b4700 2024-03-25T18:08:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.463+0000 7f25e159a640 4 set_mon_vals no callback set 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4187091609 0 0) 0x7f25d80064b0 con 0x7f25e40b4700 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 10 client.4121.objecter ms_dispatch 0x7f25e4000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 3 client.4121.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 3 client.4121.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:08:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f25d80043c0 con 0x7f25e40b4700 2024-03-25T18:08:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.464+0000 7f25cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:43.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.481+0000 7f9d64858040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:08:43.512 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.512+0000 7f887063e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:08:43.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.536+0000 7f25e90b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5134f463-b5c2-47f0-9dff-01724449ce31", "id": 0}] 2024-03-25T18:08:43.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.536+0000 7f25e90b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5134f463-b5c2-47f0-9dff-01724449ce31", "id": 0}] 2024-03-25T18:08:43.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.536+0000 7f25e90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.536+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5134f463-b5c2-47f0-9dff-01724449ce31", "id": 0} v 0) v1 -- 0x7f25e404f730 con 0x7f25e40b4700 2024-03-25T18:08:43.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.548+0000 7f9d64858040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:08:43.558 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.558+0000 7f887063e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:08:43.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.591+0000 7f9d64858040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:08:43.616 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.615+0000 7f887063e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:08:43.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25cb7fe640 1 -- 172.21.15.164:0/286989962 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5134f463-b5c2-47f0-9dff-01724449ce31", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 963360072 0 732707515) 0x7f25d80058f0 con 0x7f25e40b4700 2024-03-25T18:08:43.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5134f463-b5c2-47f0-9dff-01724449ce31", "id": 0}] 2024-03-25T18:08:43.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:43.635 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-03-25T18:08:43.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25e90b3640 10 librados: watch_flush enter 2024-03-25T18:08:43.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25e90b3640 10 librados: watch_flush exit 2024-03-25T18:08:43.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25e90b3640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:08:43.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.634+0000 7f25e90b3640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:08:43.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 10 monclient: shutdown 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 >> v1:172.21.15.46:6790/0 conn(0x7f25e40b4700 legacy=0x7f25e412ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 shutdown_connections 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 >> 172.21.15.164:0/286989962 conn(0x7f25e40668e0 msgr2=0x7f25e40b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 shutdown_connections 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 -- 172.21.15.164:0/286989962 wait complete. 2024-03-25T18:08:43.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.635+0000 7f25e90b3640 1 librados: shutdown 2024-03-25T18:08:43.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59 1 2024-03-25T18:08:43.677 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.677+0000 7f887063e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:08:43.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.728+0000 7f9d64858040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.748+0000 7f50b4b6b640 10 monclient: get_monmap_and_config 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.748+0000 7f50b4b6b640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.748+0000 7f50b4b6b640 10 monclient: monmap: 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:43.750019+0000 2024-03-25T18:08:43.749 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:43.750019+0000 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:43.750 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 1 Processor -- start 2024-03-25T18:08:43.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 1 -- start start 2024-03-25T18:08:43.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient: init 2024-03-25T18:08:43.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f50b00b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f50b0142a20 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f50b00a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00a9d60 con 0x7f50b0142a20 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b013ff10 con 0x7f50b00b4700 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00aa1f0 con 0x7f50b00a96b0 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.749+0000 7f50b4b6b640 10 monclient(hunting): authenticate will time out at 7247.045412s 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.750+0000 7f50ae59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f50b00b4700 0x7f50b0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59302/0 (socket says 172.21.15.164:59302) 2024-03-25T18:08:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.750+0000 7f50ae59c640 1 -- 172.21.15.164:0/2871171874 learned_addr learned my addr 172.21.15.164:0/2871171874 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551451693 0 0) 0x7f50b013ff10 con 0x7f50b00b4700 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f509c003520 con 0x7f50b00b4700 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310424869 0 0) 0x7f50b00a9d60 con 0x7f50b0142a20 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b013ff10 con 0x7f50b0142a20 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290499362 0 0) 0x7f50b00aa1f0 con 0x7f50b00a96b0 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:08:43.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.752+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b00a9d60 con 0x7f50b00a96b0 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2961101900 0 0) 0x7f509c003520 con 0x7f50b00b4700 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 >> v1:172.21.15.46:6790/0 conn(0x7f50b00a96b0 legacy=0x7f50b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 >> v1:172.21.15.46:6789/0 conn(0x7f50b0142a20 legacy=0x7f50b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 monclient: found mon.noname-c 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00b4ae0 con 0x7f50b00b4700 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f50a0001690 con 0x7f50b00b4700 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50b4b6b640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:08:43.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50add9b640 10 monclient: dump: 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.755+0000 7f50b4b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:43.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a00017f0 con 0x7f50b00b4700 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 1 -- 172.21.15.164:0/2871171874 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f50a0001c00 con 0x7f50b00b4700 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50add9b640 10 monclient: dump: 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.783 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50b4b6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:43.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.781+0000 7f50b4b6b640 4 set_mon_vals no callback set 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 10 monclient: shutdown 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 1 -- 172.21.15.164:0/2871171874 >> v1:172.21.15.164:6789/0 conn(0x7f50b00b4700 legacy=0x7f50b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 1 -- 172.21.15.164:0/2871171874 shutdown_connections 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 1 -- 172.21.15.164:0/2871171874 >> 172.21.15.164:0/2871171874 conn(0x7f50b00668e0 msgr2=0x7f50b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 1 -- 172.21.15.164:0/2871171874 shutdown_connections 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.782+0000 7f50b4b6b640 1 -- 172.21.15.164:0/2871171874 wait complete. 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.783+0000 7f50b4b6b640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.783+0000 7f50b4b6b640 10 monclient: monmap: 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:43.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.783+0000 7f50b4b6b640 1 librados: starting msgr at 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.783+0000 7f50b4b6b640 1 librados: starting objecter 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 Processor -- start 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 -- start start 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 librados: setting wanted keys 2024-03-25T18:08:43.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 librados: calling monclient init 2024-03-25T18:08:43.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient: init 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f50b00a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f50b00b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f50b0142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b0139660 con 0x7f50b00b4700 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b013a6e0 con 0x7f50b0142a20 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b01398d0 con 0x7f50b00a96b0 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.784+0000 7f50b4b6b640 10 monclient(hunting): authenticate will time out at 7247.080624s 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50aed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f50b0142a20 0x7f50b01e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59306/0 (socket says 172.21.15.164:59306) 2024-03-25T18:08:43.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50aed9d640 1 -- 172.21.15.164:0/4002530452 learned_addr learned my addr 172.21.15.164:0/4002530452 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50a77fe640 10 client.?.objecter ms_handle_connect 0x7f50b0142a20 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50a77fe640 10 client.?.objecter ms_handle_connect 0x7f50b00a96b0 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.785+0000 7f50a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.786+0000 7f50a77fe640 10 client.?.objecter ms_handle_connect 0x7f50b00b4700 2024-03-25T18:08:43.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.786+0000 7f50a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826172117 0 0) 0x7f50b013a6e0 con 0x7f50b0142a20 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.787+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5090004eb0 con 0x7f50b0142a20 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541565532 0 0) 0x7f50b0139660 con 0x7f50b00b4700 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b013a6e0 con 0x7f50b00b4700 2024-03-25T18:08:43.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046430037 0 0) 0x7f50b01398d0 con 0x7f50b00a96b0 2024-03-25T18:08:43.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:08:43.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.788+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0139660 con 0x7f50b00a96b0 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276312585 0 0) 0x7f50b0139660 con 0x7f50b00a96b0 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b01398d0 con 0x7f50b00a96b0 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f50a0001b00 con 0x7f50b00a96b0 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 10 monclient(hunting): dump: 2024-03-25T18:08:43.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.792+0000 7f50a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67675884 0 0) 0x7f5090004eb0 con 0x7f50b0142a20 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b0139660 con 0x7f50b0142a20 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f50a8001c70 con 0x7f50b0142a20 2024-03-25T18:08:43.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.793+0000 7f50a77fe640 10 monclient(hunting): dump: 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.796 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604952726 0 0) 0x7f50b01398d0 con 0x7f50b00a96b0 2024-03-25T18:08:43.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 >> v1:172.21.15.164:6789/0 conn(0x7f50b0142a20 legacy=0x7f50b01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 >> v1:172.21.15.46:6789/0 conn(0x7f50b00b4700 legacy=0x7f50b0139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 1 monclient: found mon.c 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0139bb0 con 0x7f50b00a96b0 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 10 monclient: _renew_subs 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0138900 con 0x7f50b00a96b0 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 10 monclient: _renew_subs 2024-03-25T18:08:43.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b0135ab0 con 0x7f50b00a96b0 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.796+0000 7f50b4b6b640 1 librados: init done 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.797+0000 7f50b4b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.797+0000 7f50b4b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.797+0000 7f50b4b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.797+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b004f730 con 0x7f50b00a96b0 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a0001c60 con 0x7f50b00a96b0 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f50a0004c30 con 0x7f50b00a96b0 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 10 monclient: dump: 2024-03-25T18:08:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f50a0005120 con 0x7f50b00a96b0 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50ad59a640 4 set_mon_vals no callback set 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3083405673 0 0) 0x7f50a0006660 con 0x7f50b00a96b0 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 10 client.4127.objecter ms_dispatch 0x7f50b0000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 3 client.4127.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:08:43.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 3 client.4127.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:08:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.800+0000 7f50a77fe640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T18:08:43.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.804+0000 7f9d64858040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:43.809 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.808+0000 7f887063e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:08:43.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.851+0000 7f9d64858040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:08:43.882 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.881+0000 7f887063e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:43.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.899+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f50a00069a0 con 0x7f50b00a96b0 2024-03-25T18:08:43.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.899+0000 7f50a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.899+0000 7f50a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:43.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.903+0000 7f9d64858040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:08:43.930 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.929+0000 7f887063e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:08:43.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:43.950+0000 7f9d64858040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:08:43.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.972+0000 7f50b4b6b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59", "id": 1}] 2024-03-25T18:08:43.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.972+0000 7f50b4b6b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59", "id": 1}] 2024-03-25T18:08:43.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.972+0000 7f50b4b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:43.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:43.972+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd new", "uuid": "1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59", "id": 1} v 0) v1 -- 0x7f50b013e650 con 0x7f50b00a96b0 2024-03-25T18:08:43.997 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:43.996+0000 7f887063e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:08:44.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.025+0000 7f50a77fe640 1 -- 172.21.15.164:0/4002530452 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3319493234 0 940483276) 0x7f50a0006b40 con 0x7f50b00a96b0 2024-03-25T18:08:44.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.025+0000 7f50a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59", "id": 1}] 2024-03-25T18:08:44.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.025+0000 7f50a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:44.027 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 10 librados: watch_flush enter 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 10 librados: watch_flush exit 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 10 client.4127.objecter successfully canceled tick 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 10 monclient: shutdown 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 >> v1:172.21.15.46:6790/0 conn(0x7f50b00a96b0 legacy=0x7f50b0138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 shutdown_connections 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 >> 172.21.15.164:0/4002530452 conn(0x7f50b00668e0 msgr2=0x7f50b00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 shutdown_connections 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 -- 172.21.15.164:0/4002530452 wait complete. 2024-03-25T18:08:44.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.026+0000 7f50b4b6b640 1 librados: shutdown 2024-03-25T18:08:44.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406 2 2024-03-25T18:08:44.059 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.059+0000 7f887063e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:08:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.138+0000 7f28976e0640 10 monclient: get_monmap_and_config 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.138+0000 7f28976e0640 10 monclient: build_initial_monmap 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.138+0000 7f28976e0640 10 monclient: monmap: 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:44.140248+0000 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:44.140248+0000 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:44.140 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.139+0000 7f28976e0640 1 Processor -- start 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.139+0000 7f28976e0640 1 -- start start 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.139+0000 7f28976e0640 10 monclient: init 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-b con 0x7f28900b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-c con 0x7f2890142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-a con 0x7f28900a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28900a9d60 con 0x7f28900a96b0 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289013ff10 con 0x7f2890142a20 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28900aa1f0 con 0x7f28900b4700 2024-03-25T18:08:44.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:44.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f28976e0640 10 monclient(hunting): authenticate will time out at 7247.435841s 2024-03-25T18:08:44.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f2894c7b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2890142a20 0x7f289013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59318/0 (socket says 172.21.15.164:59318) 2024-03-25T18:08:44.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.140+0000 7f2894c7b640 1 -- 172.21.15.164:0/4259631385 learned_addr learned my addr 172.21.15.164:0/4259631385 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213592532 0 0) 0x7f28900a9d60 con 0x7f28900a96b0 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:08:44.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.143+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f287c003520 con 0x7f28900a96b0 2024-03-25T18:08:44.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988048511 0 0) 0x7f289013ff10 con 0x7f2890142a20 2024-03-25T18:08:44.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:08:44.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.144+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28900a9d60 con 0x7f2890142a20 2024-03-25T18:08:44.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118421797 0 0) 0x7f28900aa1f0 con 0x7f28900b4700 2024-03-25T18:08:44.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:08:44.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.145+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289013ff10 con 0x7f28900b4700 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870123937 0 0) 0x7f287c003520 con 0x7f28900a96b0 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 >> v1:172.21.15.46:6790/0 conn(0x7f28900b4700 legacy=0x7f2890142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 >> v1:172.21.15.164:6789/0 conn(0x7f2890142a20 legacy=0x7f289013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 monclient: found mon.noname-a 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28900b4ae0 con 0x7f28900a96b0 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: _finish_auth 0 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:44.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f288c001a10 con 0x7f28900a96b0 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f2887fff640 10 monclient: dump: 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f28976e0640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:08:44.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.146+0000 7f28976e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f288c001b70 con 0x7f28900a96b0 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 1 -- 172.21.15.164:0/4259631385 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f288c001f80 con 0x7f28900a96b0 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f2887fff640 10 monclient: dump: 2024-03-25T18:08:44.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f28976e0640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:44.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.174+0000 7f28976e0640 4 set_mon_vals no callback set 2024-03-25T18:08:44.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 10 monclient: shutdown 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 1 -- 172.21.15.164:0/4259631385 >> v1:172.21.15.46:6789/0 conn(0x7f28900a96b0 legacy=0x7f289013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 1 -- 172.21.15.164:0/4259631385 shutdown_connections 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 1 -- 172.21.15.164:0/4259631385 >> 172.21.15.164:0/4259631385 conn(0x7f28900668e0 msgr2=0x7f2890066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 1 -- 172.21.15.164:0/4259631385 shutdown_connections 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.175+0000 7f28976e0640 1 -- 172.21.15.164:0/4259631385 wait complete. 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient: build_initial_monmap 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient: monmap: 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:44.179 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 librados: starting msgr at 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 librados: starting objecter 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 Processor -- start 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 -- start start 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 librados: setting wanted keys 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 librados: calling monclient init 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient: init 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-c con 0x7f28900a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-b con 0x7f28900b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 10 monclient(hunting): picked mon.noname-a con 0x7f2890142a20 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:44.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28901329f0 con 0x7f2890142a20 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.176+0000 7f28976e0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289013c9d0 con 0x7f28900b4700 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f28976e0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289013ccb0 con 0x7f28900a96b0 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f28976e0640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f28976e0640 10 monclient(hunting): authenticate will time out at 7247.472659s 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f2894c7b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f28900b4700 0x7f2890131bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59320/0 (socket says 172.21.15.164:59320) 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f2894c7b640 1 -- 172.21.15.164:0/2303346589 learned_addr learned my addr 172.21.15.164:0/2303346589 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f2885ffb640 10 client.?.objecter ms_handle_connect 0x7f28900b4700 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.177+0000 7f2885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.178+0000 7f2885ffb640 10 client.?.objecter ms_handle_connect 0x7f28900a96b0 2024-03-25T18:08:44.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.178+0000 7f2885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290320719 0 0) 0x7f289013c9d0 con 0x7f28900b4700 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2874004eb0 con 0x7f28900b4700 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063575741 0 0) 0x7f289013ccb0 con 0x7f28900a96b0 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:08:44.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.180+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289013c9d0 con 0x7f28900a96b0 2024-03-25T18:08:44.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.182+0000 7f2885ffb640 10 client.?.objecter ms_handle_connect 0x7f2890142a20 2024-03-25T18:08:44.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.182+0000 7f2885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733871414 0 0) 0x7f2874004eb0 con 0x7f28900b4700 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f289013ccb0 con 0x7f28900b4700 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f28800019f0 con 0x7f28900b4700 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.187 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888308112 0 0) 0x7f28901329f0 con 0x7f2890142a20 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:08:44.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.185+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2874004eb0 con 0x7f2890142a20 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1687182443 0 0) 0x7f289013c9d0 con 0x7f28900a96b0 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28901329f0 con 0x7f28900a96b0 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f28880018b0 con 0x7f28900a96b0 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.186+0000 7f2885ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.189 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1236167287 0 0) 0x7f289013ccb0 con 0x7f28900b4700 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 >> v1:172.21.15.46:6790/0 conn(0x7f28900a96b0 legacy=0x7f2890138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 >> v1:172.21.15.46:6789/0 conn(0x7f2890142a20 legacy=0x7f28901322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 1 monclient: found mon.b 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890138c50 con 0x7f28900b4700 2024-03-25T18:08:44.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f2885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f28976e0640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f28976e0640 10 monclient: _renew_subs 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f28976e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.189+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2890138ec0 con 0x7f28900b4700 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 monclient: _renew_subs 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f289013bc10 con 0x7f28900b4700 2024-03-25T18:08:44.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 1 librados: init done 2024-03-25T18:08:44.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.190+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289013b6a0 con 0x7f28900b4700 2024-03-25T18:08:44.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.354+0000 7f9d64858040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28800047e0 con 0x7f28900b4700 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2880004af0 con 0x7f28900b4700 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: dump: 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.389 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2880005020 con 0x7f28900b4700 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f28877fe640 4 set_mon_vals no callback set 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2481439593 0 0) 0x7f2880006600 con 0x7f28900b4700 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 client.4129.objecter ms_dispatch 0x7f2890000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2880006980 con 0x7f28900b4700 2024-03-25T18:08:44.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.387+0000 7f2885ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:44.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.460+0000 7f28976e0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406", "id": 2}] 2024-03-25T18:08:44.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.460+0000 7f28976e0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406", "id": 2}] 2024-03-25T18:08:44.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.460+0000 7f28976e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.460+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406", "id": 2} v 0) v1 -- 0x7f289013f7b0 con 0x7f28900b4700 2024-03-25T18:08:44.465 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.465+0000 7f887063e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:08:44.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.495+0000 7f9d64858040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:08:44.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.568+0000 7f2885ffb640 1 -- 172.21.15.164:0/2303346589 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3166473187 0 216599125) 0x7f28800062b0 con 0x7f28900b4700 2024-03-25T18:08:44.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.569+0000 7f2885ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406", "id": 2}] 2024-03-25T18:08:44.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.569+0000 7f2885ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:44.571 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-03-25T18:08:44.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 10 librados: watch_flush enter 2024-03-25T18:08:44.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 10 librados: watch_flush exit 2024-03-25T18:08:44.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:08:44.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 10 monclient: shutdown 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 >> v1:172.21.15.164:6789/0 conn(0x7f28900b4700 legacy=0x7f2890131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 shutdown_connections 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 >> 172.21.15.164:0/2303346589 conn(0x7f28900668e0 msgr2=0x7f289013e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 shutdown_connections 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 -- 172.21.15.164:0/2303346589 wait complete. 2024-03-25T18:08:44.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.570+0000 7f28976e0640 1 librados: shutdown 2024-03-25T18:08:44.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 4f499a38-37ef-4a56-8e81-2f8b72374926 3 2024-03-25T18:08:44.609 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.609+0000 7f887063e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:08:44.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.653+0000 7f9d64858040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.679+0000 7ff40fb3b640 10 monclient: get_monmap_and_config 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.679+0000 7ff40fb3b640 10 monclient: build_initial_monmap 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.679+0000 7ff40fb3b640 10 monclient: monmap: 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.680 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:44.681037+0000 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:44.681037+0000 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:44.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 1 Processor -- start 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 1 -- start start 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient: init 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4080b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff408142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4080a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080a9d60 con 0x7ff4080a96b0 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40813ff10 con 0x7ff408142a20 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080aa1f0 con 0x7ff4080b4700 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.680+0000 7ff40fb3b640 10 monclient(hunting): authenticate will time out at 7247.976559s 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.681+0000 7ff40d0d6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff408142a20 0x7ff40813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59326/0 (socket says 172.21.15.164:59326) 2024-03-25T18:08:44.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.681+0000 7ff40d0d6640 1 -- 172.21.15.164:0/3075131740 learned_addr learned my addr 172.21.15.164:0/3075131740 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526172036 0 0) 0x7ff40813ff10 con 0x7ff408142a20 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.683+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4003520 con 0x7ff408142a20 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923504581 0 0) 0x7ff4080aa1f0 con 0x7ff4080b4700 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40813ff10 con 0x7ff4080b4700 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021119952 0 0) 0x7ff4080a9d60 con 0x7ff4080a96b0 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:08:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.684+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080aa1f0 con 0x7ff4080a96b0 2024-03-25T18:08:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633545251 0 0) 0x7ff3f4003520 con 0x7ff408142a20 2024-03-25T18:08:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 >> v1:172.21.15.46:6790/0 conn(0x7ff4080b4700 legacy=0x7ff408142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 >> v1:172.21.15.46:6789/0 conn(0x7ff4080a96b0 legacy=0x7ff40813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: _finish_hunting 0 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 monclient: found mon.noname-c 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080b4ae0 con 0x7ff408142a20 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: _finish_auth 0 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: _check_auth_tickets 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff3f8001690 con 0x7ff408142a20 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40fb3b640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40fb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40c8d5640 10 monclient: dump: 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.689 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.687+0000 7ff40fb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:44.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.688+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f80017f0 con 0x7ff408142a20 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.688+0000 7ff40c8d5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.688+0000 7ff40fb3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.688+0000 7ff40fb3b640 4 set_mon_vals no callback set 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40c8d5640 1 -- 172.21.15.164:0/3075131740 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff3f8001c00 con 0x7ff408142a20 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40c8d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40c8d5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40c8d5640 10 monclient: dump: 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.693 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 10 monclient: shutdown 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3075131740 >> v1:172.21.15.164:6789/0 conn(0x7ff408142a20 legacy=0x7ff40813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3075131740 shutdown_connections 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3075131740 >> 172.21.15.164:0/3075131740 conn(0x7ff4080668e0 msgr2=0x7ff408066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3075131740 shutdown_connections 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.689+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3075131740 wait complete. 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 10 monclient: build_initial_monmap 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 10 monclient: monmap: 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:44.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 librados: starting msgr at 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 librados: starting objecter 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 Processor -- start 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 -- start start 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 librados: setting wanted keys 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 1 librados: calling monclient init 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.690+0000 7ff40fb3b640 10 monclient: init 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4080a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4080b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff408142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080abb60 con 0x7ff4080a96b0 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080abdb0 con 0x7ff4080b4700 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40814e850 con 0x7ff408142a20 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40fb3b640 10 monclient(hunting): authenticate will time out at 7247.986925s 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40d0d6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff4080b4700 0x7ff40813bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59328/0 (socket says 172.21.15.164:59328) 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff40d0d6640 1 -- 172.21.15.164:0/3519339002 learned_addr learned my addr 172.21.15.164:0/3519339002 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff3fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff4080b4700 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.691+0000 7ff3fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.692+0000 7ff3fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff4080a96b0 2024-03-25T18:08:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.692+0000 7ff3fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.692+0000 7ff3fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff408142a20 2024-03-25T18:08:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.692+0000 7ff3fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.693+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154409959 0 0) 0x7ff4080abdb0 con 0x7ff4080b4700 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec004eb0 con 0x7ff4080b4700 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630421117 0 0) 0x7ff4080abb60 con 0x7ff4080a96b0 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080abdb0 con 0x7ff4080a96b0 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700131878 0 0) 0x7ff40814e850 con 0x7ff408142a20 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:08:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.694+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080abb60 con 0x7ff408142a20 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430008177 0 0) 0x7ff3ec004eb0 con 0x7ff4080b4700 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff40814e850 con 0x7ff4080b4700 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff3f8001bc0 con 0x7ff4080b4700 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.698+0000 7ff3fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009211713 0 0) 0x7ff4080abdb0 con 0x7ff4080a96b0 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ec004eb0 con 0x7ff4080a96b0 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff404001970 con 0x7ff4080a96b0 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66211327 0 0) 0x7ff4080abb60 con 0x7ff408142a20 2024-03-25T18:08:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4080abdb0 con 0x7ff408142a20 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff400001c70 con 0x7ff408142a20 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.699+0000 7ff3fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2637193127 0 0) 0x7ff40814e850 con 0x7ff4080b4700 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 >> v1:172.21.15.46:6790/0 conn(0x7ff408142a20 legacy=0x7ff4081e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 >> v1:172.21.15.46:6789/0 conn(0x7ff4080a96b0 legacy=0x7ff40813b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 1 monclient: found mon.b 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff408139be0 con 0x7ff4080b4700 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff3fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 10 monclient: _renew_subs 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40813c9c0 con 0x7ff4080b4700 2024-03-25T18:08:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 10 monclient: _renew_subs 2024-03-25T18:08:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4080ac690 con 0x7ff4080b4700 2024-03-25T18:08:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.702+0000 7ff40fb3b640 1 librados: init done 2024-03-25T18:08:44.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.702+0000 7f9d64858040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:08:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.703+0000 7ff40fb3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.703+0000 7ff40fb3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.703+0000 7ff40fb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.703+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4080b76e0 con 0x7ff4080b4700 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.706+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f80011f0 con 0x7ff4080b4700 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff3f8004c30 con 0x7ff4080b4700 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 10 monclient: dump: 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fffff640 4 set_mon_vals no callback set 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff3f8005160 con 0x7ff4080b4700 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3957365354 0 0) 0x7ff3f8006820 con 0x7ff4080b4700 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 10 client.4135.objecter ms_dispatch 0x7ff408000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:08:44.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.707+0000 7ff3fe7fc640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T18:08:44.732 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.732+0000 7f887063e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:08:44.779 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.779+0000 7f887063e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:08:44.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.802+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff3f8006440 con 0x7ff4080b4700 2024-03-25T18:08:44.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.802+0000 7ff3fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:44.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.802+0000 7ff3fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:44.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.819+0000 7f9d64858040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.875+0000 7ff40fb3b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f499a38-37ef-4a56-8e81-2f8b72374926", "id": 3}] 2024-03-25T18:08:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.875+0000 7ff40fb3b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f499a38-37ef-4a56-8e81-2f8b72374926", "id": 3}] 2024-03-25T18:08:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.875+0000 7ff40fb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.875+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4f499a38-37ef-4a56-8e81-2f8b72374926", "id": 3} v 0) v1 -- 0x7ff40804f730 con 0x7ff4080b4700 2024-03-25T18:08:44.896 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:44.895+0000 7f887063e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:44.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:44.918+0000 7f9d64858040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:08:44.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.960+0000 7ff3fe7fc640 1 -- 172.21.15.164:0/3519339002 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f499a38-37ef-4a56-8e81-2f8b72374926", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1455869655 0 525046306) 0x7ff3f8006a30 con 0x7ff4080b4700 2024-03-25T18:08:44.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.960+0000 7ff3fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f499a38-37ef-4a56-8e81-2f8b72374926", "id": 3}] 2024-03-25T18:08:44.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.960+0000 7ff3fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:44.962 INFO:teuthology.orchestra.run.smithi164.stdout:3 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 10 librados: watch_flush enter 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 10 librados: watch_flush exit 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 10 client.4135.objecter successfully canceled tick 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 10 monclient: shutdown 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 >> v1:172.21.15.164:6789/0 conn(0x7ff4080b4700 legacy=0x7ff40813bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 shutdown_connections 2024-03-25T18:08:44.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.961+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 >> 172.21.15.164:0/3519339002 conn(0x7ff4080668e0 msgr2=0x7ff4080b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:44.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.962+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 shutdown_connections 2024-03-25T18:08:44.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.962+0000 7ff40fb3b640 1 -- 172.21.15.164:0/3519339002 wait complete. 2024-03-25T18:08:44.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:44.962+0000 7ff40fb3b640 1 librados: shutdown 2024-03-25T18:08:44.974 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 0b1ce7be-c1d6-4261-8986-44f609df1cbd 4 2024-03-25T18:08:45.013 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:45.013+0000 7f887063e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.072+0000 7f993635d640 10 monclient: get_monmap_and_config 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.072+0000 7f993635d640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.072+0000 7f993635d640 10 monclient: monmap: 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:45.074024+0000 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:45.074024+0000 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:45.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:45.075 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 1 Processor -- start 2024-03-25T18:08:45.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 1 -- start start 2024-03-25T18:08:45.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient: init 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient(hunting): picked mon.noname-c con 0x7f99300b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient(hunting): picked mon.noname-b con 0x7f9930142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient(hunting): picked mon.noname-a con 0x7f99300a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300a9d60 con 0x7f99300a96b0 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993013ff10 con 0x7f99300b4700 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300aa1f0 con 0x7f9930142a20 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.073+0000 7f993635d640 10 monclient(hunting): authenticate will time out at 7248.369546s 2024-03-25T18:08:45.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.074+0000 7f992ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f99300b4700 0x7f9930142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59344/0 (socket says 172.21.15.164:59344) 2024-03-25T18:08:45.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.074+0000 7f992ffff640 1 -- 172.21.15.164:0/821656222 learned_addr learned my addr 172.21.15.164:0/821656222 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109993717 0 0) 0x7f993013ff10 con 0x7f99300b4700 2024-03-25T18:08:45.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:08:45.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.076+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918003520 con 0x7f99300b4700 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581477690 0 0) 0x7f99300a9d60 con 0x7f99300a96b0 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993013ff10 con 0x7f99300a96b0 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107148012 0 0) 0x7f99300aa1f0 con 0x7f9930142a20 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:08:45.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.077+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300a9d60 con 0x7f9930142a20 2024-03-25T18:08:45.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873445585 0 0) 0x7f9918003520 con 0x7f99300b4700 2024-03-25T18:08:45.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 >> v1:172.21.15.46:6790/0 conn(0x7f9930142a20 legacy=0x7f993013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 >> v1:172.21.15.46:6789/0 conn(0x7f99300a96b0 legacy=0x7f993013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 monclient: found mon.noname-c 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300b4ae0 con 0x7f99300b4700 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f993635d640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f993635d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9920001690 con 0x7f99300b4700 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f992effd640 10 monclient: dump: 2024-03-25T18:08:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.080+0000 7f993635d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99200017f0 con 0x7f99300b4700 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 1 -- 172.21.15.164:0/821656222 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9920001c00 con 0x7f99300b4700 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f992effd640 10 monclient: dump: 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f993635d640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.081+0000 7f993635d640 4 set_mon_vals no callback set 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 10 monclient: shutdown 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 1 -- 172.21.15.164:0/821656222 >> v1:172.21.15.164:6789/0 conn(0x7f99300b4700 legacy=0x7f9930142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 1 -- 172.21.15.164:0/821656222 shutdown_connections 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 1 -- 172.21.15.164:0/821656222 >> 172.21.15.164:0/821656222 conn(0x7f99300668e0 msgr2=0x7f9930066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 1 -- 172.21.15.164:0/821656222 shutdown_connections 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.082+0000 7f993635d640 1 -- 172.21.15.164:0/821656222 wait complete. 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 10 monclient: monmap: 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 librados: starting msgr at 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 librados: starting objecter 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 Processor -- start 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 -- start start 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 librados: setting wanted keys 2024-03-25T18:08:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 1 librados: calling monclient init 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.083+0000 7f993635d640 10 monclient: init 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient(hunting): picked mon.noname-c con 0x7f99300a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient(hunting): picked mon.noname-a con 0x7f99300b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient(hunting): picked mon.noname-b con 0x7f9930142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300b1850 con 0x7f99300b4700 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300ad150 con 0x7f9930142a20 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300ad430 con 0x7f99300a96b0 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f993635d640 10 monclient(hunting): authenticate will time out at 7248.379859s 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f99348fa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9930142a20 0x7f99301e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59352/0 (socket says 172.21.15.164:59352) 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f99348fa640 1 -- 172.21.15.164:0/835395906 learned_addr learned my addr 172.21.15.164:0/835395906 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f992cff9640 10 client.?.objecter ms_handle_connect 0x7f9930142a20 2024-03-25T18:08:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.084+0000 7f992cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.085+0000 7f992cff9640 10 client.?.objecter ms_handle_connect 0x7f99300a96b0 2024-03-25T18:08:45.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.085+0000 7f992cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639350159 0 0) 0x7f99300ad150 con 0x7f9930142a20 2024-03-25T18:08:45.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.086+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c004eb0 con 0x7f9930142a20 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568748297 0 0) 0x7f99300ad430 con 0x7f99300a96b0 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:08:45.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300ad150 con 0x7f99300a96b0 2024-03-25T18:08:45.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:45.087+0000 7f9d64858040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:45.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.088+0000 7f992cff9640 10 client.?.objecter ms_handle_connect 0x7f99300b4700 2024-03-25T18:08:45.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.088+0000 7f992cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348474420 0 0) 0x7f99300b1850 con 0x7f99300b4700 2024-03-25T18:08:45.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300ad430 con 0x7f99300b4700 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146339106 0 0) 0x7f991c004eb0 con 0x7f9930142a20 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99300b1850 con 0x7f9930142a20 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9928001c90 con 0x7f9930142a20 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): dump: 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.093 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.091+0000 7f992cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4166942373 0 0) 0x7f99300ad150 con 0x7f99300a96b0 2024-03-25T18:08:45.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f991c004eb0 con 0x7f99300a96b0 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9920001bc0 con 0x7f99300a96b0 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.093+0000 7f992cff9640 10 monclient(hunting): dump: 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2959666083 0 0) 0x7f99300b1850 con 0x7f9930142a20 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 >> v1:172.21.15.46:6790/0 conn(0x7f99300a96b0 legacy=0x7f9930139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 >> v1:172.21.15.46:6789/0 conn(0x7f99300b4700 legacy=0x7f99300ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 1 monclient: found mon.b 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300ac180 con 0x7f9930142a20 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f992cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 10 monclient: _renew_subs 2024-03-25T18:08:45.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993013b5c0 con 0x7f9930142a20 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 10 monclient: _renew_subs 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99300abaf0 con 0x7f9930142a20 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.095+0000 7f993635d640 1 librados: init done 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.096+0000 7f993635d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.096+0000 7f993635d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.096+0000 7f993635d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.096+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993004f730 con 0x7f9930142a20 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9928003970 con 0x7f9930142a20 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9928003d00 con 0x7f9930142a20 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 10 monclient: dump: 2024-03-25T18:08:45.100 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f99280042b0 con 0x7f9930142a20 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.098+0000 7f992e7fc640 4 set_mon_vals no callback set 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.099+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1139661166 0 0) 0x7f99280069c0 con 0x7f9930142a20 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.099+0000 7f992cff9640 10 client.4141.objecter ms_dispatch 0x7f9930000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.099+0000 7f992cff9640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.099+0000 7f992cff9640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:08:45.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.099+0000 7f992cff9640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:08:45.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:45.132+0000 7f9d64858040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:45.185 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:45.184+0000 7f887063e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:08:45.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.201+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f99280065e0 con 0x7f9930142a20 2024-03-25T18:08:45.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.201+0000 7f992cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.201+0000 7f992cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:45.229 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:45.229+0000 7f887063e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:08:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.275+0000 7f993635d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0b1ce7be-c1d6-4261-8986-44f609df1cbd", "id": 4}] 2024-03-25T18:08:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.275+0000 7f993635d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0b1ce7be-c1d6-4261-8986-44f609df1cbd", "id": 4}] 2024-03-25T18:08:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.275+0000 7f993635d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.275+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0b1ce7be-c1d6-4261-8986-44f609df1cbd", "id": 4} v 0) v1 -- 0x7f99300b7190 con 0x7f9930142a20 2024-03-25T18:08:45.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:45.290+0000 7f9d64858040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:08:45.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.339+0000 7f992cff9640 1 -- 172.21.15.164:0/835395906 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0b1ce7be-c1d6-4261-8986-44f609df1cbd", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2448972735 0 1697012583) 0x7f9928006bd0 con 0x7f9930142a20 2024-03-25T18:08:45.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.339+0000 7f992cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0b1ce7be-c1d6-4261-8986-44f609df1cbd", "id": 4}] 2024-03-25T18:08:45.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.339+0000 7f992cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:45.341 INFO:teuthology.orchestra.run.smithi164.stdout:4 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 10 librados: watch_flush enter 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 10 librados: watch_flush exit 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 10 monclient: shutdown 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 >> v1:172.21.15.164:6789/0 conn(0x7f9930142a20 legacy=0x7f99301e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 shutdown_connections 2024-03-25T18:08:45.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.340+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 >> 172.21.15.164:0/835395906 conn(0x7f99300668e0 msgr2=0x7f99301364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.341+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 shutdown_connections 2024-03-25T18:08:45.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.341+0000 7f993635d640 1 -- 172.21.15.164:0/835395906 wait complete. 2024-03-25T18:08:45.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.341+0000 7f993635d640 1 librados: shutdown 2024-03-25T18:08:45.352 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new f362b74d-5f3b-4df8-bbe5-687f796063c9 5 2024-03-25T18:08:45.397 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:45.397+0000 7f887063e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:08:45.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.451+0000 7f1c3a723640 10 monclient: get_monmap_and_config 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.451+0000 7f1c3a723640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.451+0000 7f1c3a723640 10 monclient: monmap: 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:45.453083+0000 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:45.453083+0000 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:45.453 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 1 Processor -- start 2024-03-25T18:08:45.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 1 -- start start 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient: init 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-a con 0x7f1c340b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-b con 0x7f1c34142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-c con 0x7f1c340a96b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340a9d60 con 0x7f1c340b4700 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3413ff10 con 0x7f1c340a96b0 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340aa1f0 con 0x7f1c34142a20 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.452+0000 7f1c3a723640 10 monclient(hunting): authenticate will time out at 7248.748606s 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.453+0000 7f1c38cc0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1c340a96b0 0x7f1c3413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59354/0 (socket says 172.21.15.164:59354) 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.453+0000 7f1c38cc0640 1 -- 172.21.15.164:0/761979074 learned_addr learned my addr 172.21.15.164:0/761979074 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175001827 0 0) 0x7f1c3413ff10 con 0x7f1c340a96b0 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.455+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c003520 con 0x7f1c340a96b0 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455405575 0 0) 0x7f1c340a9d60 con 0x7f1c340b4700 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3413ff10 con 0x7f1c340b4700 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320210565 0 0) 0x7f1c340aa1f0 con 0x7f1c34142a20 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:08:45.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.456+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c340a9d60 con 0x7f1c34142a20 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331654632 0 0) 0x7f1c1c003520 con 0x7f1c340a96b0 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 >> v1:172.21.15.46:6790/0 conn(0x7f1c34142a20 legacy=0x7f1c3413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 >> v1:172.21.15.46:6789/0 conn(0x7f1c340b4700 legacy=0x7f1c34142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 1 monclient: found mon.noname-c 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.458+0000 7f1c32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340b4ae0 con 0x7f1c340a96b0 2024-03-25T18:08:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c2c001a10 con 0x7f1c340a96b0 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c3a723640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.459+0000 7f1c3a723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c2c001b70 con 0x7f1c340a96b0 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c3a723640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c3a723640 4 set_mon_vals no callback set 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 1 -- 172.21.15.164:0/761979074 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c2c001f80 con 0x7f1c340a96b0 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.460+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.465 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 10 monclient: shutdown 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 1 -- 172.21.15.164:0/761979074 >> v1:172.21.15.164:6789/0 conn(0x7f1c340a96b0 legacy=0x7f1c3413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 1 -- 172.21.15.164:0/761979074 shutdown_connections 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 1 -- 172.21.15.164:0/761979074 >> 172.21.15.164:0/761979074 conn(0x7f1c340668e0 msgr2=0x7f1c34066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 1 -- 172.21.15.164:0/761979074 shutdown_connections 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 1 -- 172.21.15.164:0/761979074 wait complete. 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.461+0000 7f1c3a723640 10 monclient: monmap: 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 librados: starting msgr at 2024-03-25T18:08:45.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 librados: starting objecter 2024-03-25T18:08:45.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 Processor -- start 2024-03-25T18:08:45.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 -- start start 2024-03-25T18:08:45.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 librados: setting wanted keys 2024-03-25T18:08:45.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 1 librados: calling monclient init 2024-03-25T18:08:45.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.462+0000 7f1c3a723640 10 monclient: init 2024-03-25T18:08:45.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-b con 0x7f1c340a96b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-a con 0x7f1c340b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-c con 0x7f1c34142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340ac2c0 con 0x7f1c340b4700 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c341d3fd0 con 0x7f1c340a96b0 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c341d4160 con 0x7f1c34142a20 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c3a723640 10 monclient(hunting): authenticate will time out at 7248.758740s 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c33fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1c340a96b0 0x7f1c341d7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59356/0 (socket says 172.21.15.164:59356) 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c33fff640 1 -- 172.21.15.164:0/1917217312 learned_addr learned my addr 172.21.15.164:0/1917217312 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340a96b0 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340b4700 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c34142a20 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.463+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280478140 0 0) 0x7f1c341d3fd0 con 0x7f1c340a96b0 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.465+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c340a96b0 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446355102 0 0) 0x7f1c341d4160 con 0x7f1c34142a20 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c341d3fd0 con 0x7f1c34142a20 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97996527 0 0) 0x7f1c340ac2c0 con 0x7f1c340b4700 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:08:45.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.466+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c341d4160 con 0x7f1c340b4700 2024-03-25T18:08:45.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387674598 0 0) 0x7f1c20004eb0 con 0x7f1c340a96b0 2024-03-25T18:08:45.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c340ac2c0 con 0x7f1c340a96b0 2024-03-25T18:08:45.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c28001a30 con 0x7f1c340a96b0 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.470+0000 7f1c30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387375958 0 0) 0x7f1c341d3fd0 con 0x7f1c34142a20 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c34142a20 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c2c001e80 con 0x7f1c34142a20 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.475 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635687556 0 0) 0x7f1c341d4160 con 0x7f1c340b4700 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c341d3fd0 con 0x7f1c340b4700 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c24001790 con 0x7f1c340b4700 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.471+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.476 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.477 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.477 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1040894211 0 0) 0x7f1c340ac2c0 con 0x7f1c340a96b0 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 >> v1:172.21.15.46:6790/0 conn(0x7f1c34142a20 legacy=0x7f1c3413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 >> v1:172.21.15.46:6789/0 conn(0x7f1c340b4700 legacy=0x7f1c341d47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 1 monclient: found mon.b 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c341d4ed0 con 0x7f1c340a96b0 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 10 monclient: _renew_subs 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c341d8160 con 0x7f1c340a96b0 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 10 monclient: _renew_subs 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3413c450 con 0x7f1c340a96b0 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.473+0000 7f1c3a723640 1 librados: init done 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.474+0000 7f1c3a723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.474+0000 7f1c3a723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.474+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.474+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c341d5120 con 0x7f1c340a96b0 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c28001c70 con 0x7f1c340a96b0 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c28003a60 con 0x7f1c340a96b0 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 10 monclient: dump: 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.573 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1c28003fd0 con 0x7f1c340a96b0 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c327fc640 4 set_mon_vals no callback set 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 761981549 0 0) 0x7f1c280066a0 con 0x7f1c340a96b0 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 10 client.4147.objecter ms_dispatch 0x7f1c34000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.571+0000 7f1c30ff9640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.572+0000 7f1c30ff9640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.572+0000 7f1c30ff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.572+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1c28006ae0 con 0x7f1c340a96b0 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.572+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.572+0000 7f1c30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:45.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.643+0000 7f1c3a723640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f362b74d-5f3b-4df8-bbe5-687f796063c9", "id": 5}] 2024-03-25T18:08:45.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.644+0000 7f1c3a723640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f362b74d-5f3b-4df8-bbe5-687f796063c9", "id": 5}] 2024-03-25T18:08:45.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.644+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.644+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f362b74d-5f3b-4df8-bbe5-687f796063c9", "id": 5} v 0) v1 -- 0x7f1c3404f730 con 0x7f1c340a96b0 2024-03-25T18:08:45.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.704+0000 7f1c30ff9640 1 -- 172.21.15.164:0/1917217312 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f362b74d-5f3b-4df8-bbe5-687f796063c9", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1906241826 0 1988412176) 0x7f1c28006120 con 0x7f1c340a96b0 2024-03-25T18:08:45.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.704+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f362b74d-5f3b-4df8-bbe5-687f796063c9", "id": 5}] 2024-03-25T18:08:45.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.704+0000 7f1c30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:45.706 INFO:teuthology.orchestra.run.smithi164.stdout:5 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 10 librados: watch_flush enter 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 10 librados: watch_flush exit 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 10 monclient: shutdown 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 >> v1:172.21.15.164:6789/0 conn(0x7f1c340a96b0 legacy=0x7f1c341d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 shutdown_connections 2024-03-25T18:08:45.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 >> 172.21.15.164:0/1917217312 conn(0x7f1c340668e0 msgr2=0x7f1c340b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 shutdown_connections 2024-03-25T18:08:45.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 -- 172.21.15.164:0/1917217312 wait complete. 2024-03-25T18:08:45.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.705+0000 7f1c3a723640 1 librados: shutdown 2024-03-25T18:08:45.716 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e8470de7-cbc1-44c0-8b6d-bb7d71044b72 6 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.815+0000 7f656c6ae640 10 monclient: get_monmap_and_config 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.815+0000 7f656c6ae640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.815+0000 7f656c6ae640 10 monclient: monmap: 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:45.817373+0000 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:45.817373+0000 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:45.817 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.816+0000 7f656c6ae640 1 Processor -- start 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.816+0000 7f656c6ae640 1 -- start start 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.816+0000 7f656c6ae640 10 monclient: init 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-a con 0x7f65640b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-c con 0x7f6564142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-b con 0x7f65640a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65640a9d60 con 0x7f65640b4700 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656413ff10 con 0x7f6564142a20 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65640aa1f0 con 0x7f65640a96b0 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f656c6ae640 10 monclient(hunting): authenticate will time out at 7249.112897s 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f6569c49640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6564142a20 0x7f656413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59364/0 (socket says 172.21.15.164:59364) 2024-03-25T18:08:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.817+0000 7f6569c49640 1 -- 172.21.15.164:0/3940122232 learned_addr learned my addr 172.21.15.164:0/3940122232 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574461746 0 0) 0x7f656413ff10 con 0x7f6564142a20 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.819+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6550003520 con 0x7f6564142a20 2024-03-25T18:08:45.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829185231 0 0) 0x7f65640aa1f0 con 0x7f65640a96b0 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656413ff10 con 0x7f65640a96b0 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420150127 0 0) 0x7f65640a9d60 con 0x7f65640b4700 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:08:45.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.820+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65640aa1f0 con 0x7f65640b4700 2024-03-25T18:08:45.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550902495 0 0) 0x7f6550003520 con 0x7f6564142a20 2024-03-25T18:08:45.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 >> v1:172.21.15.46:6790/0 conn(0x7f65640a96b0 legacy=0x7f656413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 >> v1:172.21.15.46:6789/0 conn(0x7f65640b4700 legacy=0x7f6564142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 monclient: found mon.noname-c 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640b4ae0 con 0x7f6564142a20 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f656c6ae640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6560001690 con 0x7f6564142a20 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f6569448640 10 monclient: dump: 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.825 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.823+0000 7f656c6ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65600017f0 con 0x7f6564142a20 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 1 -- 172.21.15.164:0/3940122232 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6560001c00 con 0x7f6564142a20 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f6569448640 10 monclient: dump: 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f656c6ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.824+0000 7f656c6ae640 4 set_mon_vals no callback set 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 10 monclient: shutdown 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 1 -- 172.21.15.164:0/3940122232 >> v1:172.21.15.164:6789/0 conn(0x7f6564142a20 legacy=0x7f656413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 1 -- 172.21.15.164:0/3940122232 shutdown_connections 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 1 -- 172.21.15.164:0/3940122232 >> 172.21.15.164:0/3940122232 conn(0x7f65640668e0 msgr2=0x7f6564066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 1 -- 172.21.15.164:0/3940122232 shutdown_connections 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.825+0000 7f656c6ae640 1 -- 172.21.15.164:0/3940122232 wait complete. 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 10 monclient: monmap: 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 librados: starting msgr at 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 librados: starting objecter 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 Processor -- start 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 -- start start 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 librados: setting wanted keys 2024-03-25T18:08:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 1 librados: calling monclient init 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.826+0000 7f656c6ae640 10 monclient: init 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-a con 0x7f65640a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-c con 0x7f65640b4700 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient(hunting): picked mon.noname-b con 0x7f6564142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656414eca0 con 0x7f65640a96b0 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65641d74d0 con 0x7f6564142a20 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65641d8350 con 0x7f65640b4700 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656c6ae640 10 monclient(hunting): authenticate will time out at 7249.122773s 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656ac4b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6564142a20 0x7f65641d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59380/0 (socket says 172.21.15.164:59380) 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f656ac4b640 1 -- 172.21.15.164:0/2859068444 learned_addr learned my addr 172.21.15.164:0/2859068444 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f655affd640 10 client.?.objecter ms_handle_connect 0x7f6564142a20 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f655affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f655affd640 10 client.?.objecter ms_handle_connect 0x7f65640b4700 2024-03-25T18:08:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.827+0000 7f655affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.828+0000 7f655affd640 10 client.?.objecter ms_handle_connect 0x7f65640a96b0 2024-03-25T18:08:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.828+0000 7f655affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823010883 0 0) 0x7f65641d74d0 con 0x7f6564142a20 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6548004eb0 con 0x7f6564142a20 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584612320 0 0) 0x7f65641d8350 con 0x7f65640b4700 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.829+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65641d74d0 con 0x7f65640b4700 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661240522 0 0) 0x7f65640aa1f0 con 0x7f65640a96b0 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:08:45.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.830+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65641d8350 con 0x7f65640a96b0 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582024542 0 0) 0x7f6548004eb0 con 0x7f6564142a20 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65640aa1f0 con 0x7f6564142a20 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f655c001c90 con 0x7f6564142a20 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): dump: 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.836 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443221369 0 0) 0x7f65641d74d0 con 0x7f65640b4700 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6548004eb0 con 0x7f65640b4700 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6560001b00 con 0x7f65640b4700 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.834+0000 7f655affd640 10 monclient(hunting): dump: 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.837 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.838 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.838 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464707959 0 0) 0x7f65641d8350 con 0x7f65640a96b0 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65641d74d0 con 0x7f65640a96b0 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6554001690 con 0x7f65640a96b0 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient(hunting): dump: 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656782346 0 0) 0x7f65640aa1f0 con 0x7f6564142a20 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 >> v1:172.21.15.46:6790/0 conn(0x7f65640b4700 legacy=0x7f65640acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 >> v1:172.21.15.46:6789/0 conn(0x7f65640a96b0 legacy=0x7f65641d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 monclient: found mon.b 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640abd90 con 0x7f6564142a20 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f655affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 10 monclient: _renew_subs 2024-03-25T18:08:45.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65641d7720 con 0x7f6564142a20 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 10 monclient: _renew_subs 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65640ad240 con 0x7f6564142a20 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.837+0000 7f656c6ae640 1 librados: init done 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.838+0000 7f656c6ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.838+0000 7f656c6ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.838+0000 7f656c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:45.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.838+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656404f730 con 0x7f6564142a20 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c003970 con 0x7f6564142a20 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f655c003d00 con 0x7f6564142a20 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 10 monclient: dump: 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:45.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f6568c47640 4 set_mon_vals no callback set 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f655c0042b0 con 0x7f6564142a20 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3328251815 0 0) 0x7f655c006b40 con 0x7f6564142a20 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 10 client.4153.objecter ms_dispatch 0x7f6564000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:08:45.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.841+0000 7f655affd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:08:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.937+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f655c004560 con 0x7f6564142a20 2024-03-25T18:08:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.937+0000 7f655affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:45.937+0000 7f655affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:46.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.010+0000 7f656c6ae640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e8470de7-cbc1-44c0-8b6d-bb7d71044b72", "id": 6}] 2024-03-25T18:08:46.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.010+0000 7f656c6ae640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e8470de7-cbc1-44c0-8b6d-bb7d71044b72", "id": 6}] 2024-03-25T18:08:46.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.010+0000 7f656c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:46.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.010+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e8470de7-cbc1-44c0-8b6d-bb7d71044b72", "id": 6} v 0) v1 -- 0x7f65641d78b0 con 0x7f6564142a20 2024-03-25T18:08:46.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.087+0000 7f655affd640 1 -- 172.21.15.164:0/2859068444 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8470de7-cbc1-44c0-8b6d-bb7d71044b72", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 8167143 0 1113811849) 0x7f655c004790 con 0x7f6564142a20 2024-03-25T18:08:46.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.087+0000 7f655affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e8470de7-cbc1-44c0-8b6d-bb7d71044b72", "id": 6}] 2024-03-25T18:08:46.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.087+0000 7f655affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:46.089 INFO:teuthology.orchestra.run.smithi164.stdout:6 2024-03-25T18:08:46.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 10 librados: watch_flush enter 2024-03-25T18:08:46.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 10 librados: watch_flush exit 2024-03-25T18:08:46.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:08:46.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 10 monclient: shutdown 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 >> v1:172.21.15.164:6789/0 conn(0x7f6564142a20 legacy=0x7f65641d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 shutdown_connections 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 >> 172.21.15.164:0/2859068444 conn(0x7f65640668e0 msgr2=0x7f65640b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 shutdown_connections 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 -- 172.21.15.164:0/2859068444 wait complete. 2024-03-25T18:08:46.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.088+0000 7f656c6ae640 1 librados: shutdown 2024-03-25T18:08:46.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e449439f-9f64-42ae-aecc-87980c327dd4 7 2024-03-25T18:08:46.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:46.123+0000 7f9d64858040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.199+0000 7fba73b64640 10 monclient: get_monmap_and_config 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.199+0000 7fba73b64640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.199+0000 7fba73b64640 10 monclient: monmap: 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:08:46.200948+0000 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:08:46.200948+0000 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 1 Processor -- start 2024-03-25T18:08:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 1 -- start start 2024-03-25T18:08:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient: init 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c0b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-b con 0x7fba6c142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-a con 0x7fba6c0a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0a9d60 con 0x7fba6c0a96b0 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c13ff10 con 0x7fba6c0b4700 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0aa1f0 con 0x7fba6c142a20 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.200+0000 7fba73b64640 10 monclient(hunting): authenticate will time out at 7249.496536s 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.201+0000 7fba710ff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fba6c142a20 0x7fba6c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.164:40082/0 (socket says 172.21.15.164:40082) 2024-03-25T18:08:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.201+0000 7fba710ff640 1 -- 172.21.15.164:0/4069409860 learned_addr learned my addr 172.21.15.164:0/4069409860 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877174161 0 0) 0x7fba6c13ff10 con 0x7fba6c0b4700 2024-03-25T18:08:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:08:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.204+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba58003520 con 0x7fba6c0b4700 2024-03-25T18:08:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321593405 0 0) 0x7fba6c0a9d60 con 0x7fba6c0a96b0 2024-03-25T18:08:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c13ff10 con 0x7fba6c0a96b0 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266425040 0 0) 0x7fba6c0aa1f0 con 0x7fba6c142a20 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:08:46.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.206+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c0a9d60 con 0x7fba6c142a20 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869049129 0 0) 0x7fba6c0a9d60 con 0x7fba6c142a20 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 >> v1:172.21.15.164:6789/0 conn(0x7fba6c0b4700 legacy=0x7fba6c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 >> v1:172.21.15.46:6789/0 conn(0x7fba6c0a96b0 legacy=0x7fba6c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 monclient: found mon.noname-b 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0b4ae0 con 0x7fba6c142a20 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba5c0015b0 con 0x7fba6c142a20 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba73b64640 5 monclient: authenticate success, global_id 4160 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba73b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba708fe640 10 monclient: dump: 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.212 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.209+0000 7fba73b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba5c001710 con 0x7fba6c142a20 2024-03-25T18:08:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 1 -- 172.21.15.164:0/4069409860 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba5c001b60 con 0x7fba6c142a20 2024-03-25T18:08:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba73b64640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba73b64640 4 set_mon_vals no callback set 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.211+0000 7fba708fe640 10 monclient: dump: 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 10 monclient: shutdown 2024-03-25T18:08:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 1 -- 172.21.15.164:0/4069409860 >> v1:172.21.15.46:6790/0 conn(0x7fba6c142a20 legacy=0x7fba6c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 1 -- 172.21.15.164:0/4069409860 shutdown_connections 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 1 -- 172.21.15.164:0/4069409860 >> 172.21.15.164:0/4069409860 conn(0x7fba6c0668e0 msgr2=0x7fba6c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 1 -- 172.21.15.164:0/4069409860 shutdown_connections 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.212+0000 7fba73b64640 1 -- 172.21.15.164:0/4069409860 wait complete. 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 10 monclient: monmap: 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 librados: starting msgr at 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 librados: starting objecter 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 Processor -- start 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 -- start start 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 librados: setting wanted keys 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 1 librados: calling monclient init 2024-03-25T18:08:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.213+0000 7fba73b64640 10 monclient: init 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-a con 0x7fba6c0a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-b con 0x7fba6c0b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c0addb0 con 0x7fba6c0a96b0 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c1db670 con 0x7fba6c0b4700 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c1db270 con 0x7fba6c142a20 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba73b64640 10 monclient(hunting): authenticate will time out at 7249.509836s 2024-03-25T18:08:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba72101640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fba6c142a20 0x7fba6c1d7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.164:40096/0 (socket says 172.21.15.164:40096) 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.214+0000 7fba72101640 1 -- 172.21.15.164:0/3930905577 learned_addr learned my addr 172.21.15.164:0/3930905577 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.215+0000 7fba627fc640 10 client.?.objecter ms_handle_connect 0x7fba6c142a20 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.215+0000 7fba627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.215+0000 7fba627fc640 10 client.?.objecter ms_handle_connect 0x7fba6c0a96b0 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.215+0000 7fba627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059417864 0 0) 0x7fba6c1db270 con 0x7fba6c142a20 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50004eb0 con 0x7fba6c142a20 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061947635 0 0) 0x7fba6c0addb0 con 0x7fba6c0a96b0 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:08:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.217+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c1db270 con 0x7fba6c0a96b0 2024-03-25T18:08:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.220+0000 7fba627fc640 10 client.?.objecter ms_handle_connect 0x7fba6c0b4700 2024-03-25T18:08:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.220+0000 7fba627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636500066 0 0) 0x7fba6c1db670 con 0x7fba6c0b4700 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c0addb0 con 0x7fba6c0b4700 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028954862 0 0) 0x7fba50004eb0 con 0x7fba6c142a20 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba6c1db670 con 0x7fba6c142a20 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba64001d10 con 0x7fba6c142a20 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): dump: 2024-03-25T18:08:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.222+0000 7fba627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578175254 0 0) 0x7fba6c1db270 con 0x7fba6c0a96b0 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba50004eb0 con 0x7fba6c0a96b0 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba68001950 con 0x7fba6c0a96b0 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.223+0000 7fba627fc640 10 monclient(hunting): dump: 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3305840146 0 0) 0x7fba6c1db670 con 0x7fba6c142a20 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 >> v1:172.21.15.164:6789/0 conn(0x7fba6c0b4700 legacy=0x7fba6c1d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 >> v1:172.21.15.46:6789/0 conn(0x7fba6c0a96b0 legacy=0x7fba6c1dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 1 monclient: found mon.c 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c1db400 con 0x7fba6c142a20 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 5 monclient: authenticate success, global_id 4163 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 10 monclient: _renew_subs 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c1e2c30 con 0x7fba6c142a20 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 10 monclient: _renew_subs 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c1e2fa0 con 0x7fba6c142a20 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.225+0000 7fba73b64640 1 librados: init done 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.226+0000 7fba73b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.226+0000 7fba73b64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.226+0000 7fba73b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.226+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c04f730 con 0x7fba6c142a20 2024-03-25T18:08:46.268 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:46.268+0000 7f887063e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:08:46.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba64003990 con 0x7fba6c142a20 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba64003d60 con 0x7fba6c142a20 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 10 monclient: dump: 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 3 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fba64004310 con 0x7fba6c142a20 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3673513655 0 0) 0x7fba64006b60 con 0x7fba6c142a20 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba63fff640 4 set_mon_vals no callback set 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 10 client.4163.objecter ms_dispatch 0x7fba6c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 3 client.4163.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 3 client.4163.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.328+0000 7fba627fc640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.330+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fba640045c0 con 0x7fba6c142a20 2024-03-25T18:08:46.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.330+0000 7fba627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.330+0000 7fba627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.401+0000 7fba73b64640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e449439f-9f64-42ae-aecc-87980c327dd4", "id": 7}] 2024-03-25T18:08:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.401+0000 7fba73b64640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e449439f-9f64-42ae-aecc-87980c327dd4", "id": 7}] 2024-03-25T18:08:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.401+0000 7fba73b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.401+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd new", "uuid": "e449439f-9f64-42ae-aecc-87980c327dd4", "id": 7} v 0) v1 -- 0x7fba6c069f30 con 0x7fba6c142a20 2024-03-25T18:08:46.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:46.410+0000 7f9d64858040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:08:46.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.429+0000 7fba627fc640 1 -- 172.21.15.164:0/3930905577 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e449439f-9f64-42ae-aecc-87980c327dd4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2626062493 0 1371665406) 0x7fba640065b0 con 0x7fba6c142a20 2024-03-25T18:08:46.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.429+0000 7fba627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e449439f-9f64-42ae-aecc-87980c327dd4", "id": 7}] 2024-03-25T18:08:46.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.429+0000 7fba627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:46.431 INFO:teuthology.orchestra.run.smithi164.stdout:7 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 10 librados: watch_flush enter 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 10 librados: watch_flush exit 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 10 client.4163.objecter successfully canceled tick 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 10 monclient: shutdown 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.430+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 >> v1:172.21.15.46:6790/0 conn(0x7fba6c142a20 legacy=0x7fba6c1d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.431+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 shutdown_connections 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.431+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 >> 172.21.15.164:0/3930905577 conn(0x7fba6c0668e0 msgr2=0x7fba6c13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.431+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 shutdown_connections 2024-03-25T18:08:46.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.431+0000 7fba73b64640 1 -- 172.21.15.164:0/3930905577 wait complete. 2024-03-25T18:08:46.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-25T18:08:46.431+0000 7fba73b64640 1 librados: shutdown 2024-03-25T18:08:46.442 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.443 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:08:46.443 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:08:46.445 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:08:46.445 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.445 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:08:46.446 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:08:46.447 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:08:46.448 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.448 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:08:46.448 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:08:46.450 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:08:46.450 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.450 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:08:46.450 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:08:46.453 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:08:46.453 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.453 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:08:46.453 DEBUG:teuthology.orchestra.run.smithi164:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:08:46.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:46.465+0000 7f9d64858040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:08:46.468 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:08:46.468 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.468 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:08:46.468 DEBUG:teuthology.orchestra.run.smithi164:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:08:46.470 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:08:46.470 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.470 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:08:46.470 DEBUG:teuthology.orchestra.run.smithi164:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:08:46.472 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:08:46.472 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:08:46.472 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:08:46.473 DEBUG:teuthology.orchestra.run.smithi164:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:08:46.475 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:08:46.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:46.614 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:46.614+0000 7f887063e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.641+0000 7f4795060640 10 monclient: get_monmap_and_config 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.641+0000 7f4795060640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.641+0000 7f4795060640 10 monclient: monmap: 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:08:46.642123+0000 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:08:46.642123+0000 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:46.642 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.642+0000 7f4795060640 1 Processor -- start 2024-03-25T18:08:46.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.642+0000 7f4795060640 1 -- start start 2024-03-25T18:08:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.642+0000 7f4795060640 10 monclient: init 2024-03-25T18:08:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.643+0000 7f4795060640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.643+0000 7f4795060640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.643+0000 7f4795060640 10 monclient(hunting): picked mon.noname-a con 0x7f47900b4700 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.643+0000 7f4795060640 10 monclient(hunting): picked mon.noname-c con 0x7f4790142a20 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:46.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:46.643+0000 7f9d64858040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:08:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 10 monclient(hunting): picked mon.noname-b con 0x7f47900a96b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900a9d60 con 0x7f47900b4700 2024-03-25T18:08:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479013ff10 con 0x7f4790142a20 2024-03-25T18:08:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900aa1f0 con 0x7f47900a96b0 2024-03-25T18:08:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f4795060640 10 monclient(hunting): authenticate will time out at 7235.905690s 2024-03-25T18:08:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f478ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f47900b4700 0x7f4790142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57264/0 (socket says 172.21.15.46:57264) 2024-03-25T18:08:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.644+0000 7f478ed9d640 1 -- 172.21.15.46:0/3867794376 learned_addr learned my addr 172.21.15.46:0/3867794376 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877241450 0 0) 0x7f479013ff10 con 0x7f4790142a20 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c003520 con 0x7f4790142a20 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991429923 0 0) 0x7f47900aa1f0 con 0x7f47900a96b0 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f479013ff10 con 0x7f47900a96b0 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330999755 0 0) 0x7f47900a9d60 con 0x7f47900b4700 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:08:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.648+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47900aa1f0 con 0x7f47900b4700 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126998634 0 0) 0x7f477c003520 con 0x7f4790142a20 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 >> v1:172.21.15.46:6790/0 conn(0x7f47900a96b0 legacy=0x7f479013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 >> v1:172.21.15.46:6789/0 conn(0x7f47900b4700 legacy=0x7f4790142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 monclient: found mon.noname-c 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900b4ae0 con 0x7f4790142a20 2024-03-25T18:08:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4778001650 con 0x7f4790142a20 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.653+0000 7f478e59c640 10 monclient: dump: 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.656 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.654+0000 7f4795060640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:08:46.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.654+0000 7f4795060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:46.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47780018a0 con 0x7f4790142a20 2024-03-25T18:08:46.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f4795060640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:46.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f4795060640 4 set_mon_vals no callback set 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 1 -- 172.21.15.46:0/3867794376 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4778001c30 con 0x7f4790142a20 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.656+0000 7f478e59c640 10 monclient: dump: 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.657+0000 7f4795060640 10 monclient: shutdown 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.657+0000 7f4795060640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.657+0000 7f4795060640 1 -- 172.21.15.46:0/3867794376 >> v1:172.21.15.164:6789/0 conn(0x7f4790142a20 legacy=0x7f479013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.657+0000 7f4795060640 1 -- 172.21.15.46:0/3867794376 shutdown_connections 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.658+0000 7f4795060640 1 -- 172.21.15.46:0/3867794376 >> 172.21.15.46:0/3867794376 conn(0x7f47900668e0 msgr2=0x7f4790066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.658+0000 7f4795060640 1 -- 172.21.15.46:0/3867794376 shutdown_connections 2024-03-25T18:08:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.658+0000 7f4795060640 1 -- 172.21.15.46:0/3867794376 wait complete. 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.659+0000 7f4795060640 10 monclient: build_initial_monmap 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.659+0000 7f4795060640 10 monclient: monmap: 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 0 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 0.000000 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:created 0.000000 2024-03-25T18:08:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 1 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 librados: starting msgr at 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 librados: starting objecter 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 Processor -- start 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 -- start start 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 librados: setting wanted keys 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 1 librados: calling monclient init 2024-03-25T18:08:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.660+0000 7f4795060640 10 monclient: init 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.661+0000 7f4795060640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 10 monclient(hunting): picked mon.noname-a con 0x7f47900a96b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 10 monclient(hunting): picked mon.noname-b con 0x7f47900b4700 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 10 monclient(hunting): picked mon.noname-c con 0x7f4790142a20 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900ac310 con 0x7f47900a96b0 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900adde0 con 0x7f47900b4700 2024-03-25T18:08:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900ae0c0 con 0x7f4790142a20 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4795060640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4787fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f47900b4700 0x7f479013aaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36684/0 (socket says 172.21.15.46:36684) 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.662+0000 7f4787fff640 1 -- 172.21.15.46:0/2899014777 learned_addr learned my addr 172.21.15.46:0/2899014777 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.663+0000 7f4795060640 10 monclient(hunting): authenticate will time out at 7235.924596s 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.663+0000 7f47877fe640 10 client.?.objecter ms_handle_connect 0x7f47900b4700 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.663+0000 7f47877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.663+0000 7f47877fe640 10 client.?.objecter ms_handle_connect 0x7f47900a96b0 2024-03-25T18:08:46.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.663+0000 7f47877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:46.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453576155 0 0) 0x7f47900adde0 con 0x7f47900b4700 2024-03-25T18:08:46.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770004eb0 con 0x7f47900b4700 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242546115 0 0) 0x7f47900ac310 con 0x7f47900a96b0 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:08:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.666+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47900adde0 con 0x7f47900a96b0 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700874910 0 0) 0x7f4770004eb0 con 0x7f47900b4700 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47900ac310 con 0x7f47900b4700 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4778001d70 con 0x7f47900b4700 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 10 monclient(hunting): dump: 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.673+0000 7f47877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194356696 0 0) 0x7f47900adde0 con 0x7f47900a96b0 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4770004eb0 con 0x7f47900a96b0 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f47800018b0 con 0x7f47900a96b0 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.674+0000 7f47877fe640 10 monclient(hunting): dump: 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.677 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495457271 0 0) 0x7f4770004eb0 con 0x7f47900a96b0 2024-03-25T18:08:46.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 >> v1:172.21.15.46:6790/0 conn(0x7f4790142a20 legacy=0x7f47901e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 >> v1:172.21.15.164:6789/0 conn(0x7f47900b4700 legacy=0x7f479013aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 1 monclient: found mon.a 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900ae3a0 con 0x7f47900a96b0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f47877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 10 monclient: _renew_subs 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47900abd40 con 0x7f47900a96b0 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 10 monclient: _renew_subs 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47900aca30 con 0x7f47900a96b0 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.679+0000 7f4795060640 1 librados: init done 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.681+0000 7f4795060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.681+0000 7f4795060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.681+0000 7f4795060640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:08:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.681+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479004f730 con 0x7f47900a96b0 2024-03-25T18:08:46.704 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:46.704+0000 7f887063e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4780001d70 con 0x7f47900a96b0 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4780004a00 con 0x7f47900a96b0 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 10 monclient: dump: 2024-03-25T18:08:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:epoch 1 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4780004fb0 con 0x7f47900a96b0 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f4780006930 con 0x7f47900a96b0 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 10 client.4158.objecter ms_dispatch 0x7f4790000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 3 client.4158.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 3 client.4158.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f47877fe640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-25T18:08:46.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:46.712+0000 7f478dd9b640 4 set_mon_vals no callback set 2024-03-25T18:08:46.918 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-03-25T18:08:46.917+0000 7f887063e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:08:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.000+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4780006b40 con 0x7f47900a96b0 2024-03-25T18:08:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.000+0000 7f47877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:47.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.000+0000 7f47877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.119+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2531371089 0 0) 0x7f4780006370 con 0x7f47900a96b0 2024-03-25T18:08:47.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.124+0000 7f4795060640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:47.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.124+0000 7f4795060640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:47.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.124+0000 7f4795060640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:08:47.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.124+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47900b7710 con 0x7f47900a96b0 2024-03-25T18:08:47.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.291+0000 7f47877fe640 1 -- 172.21.15.46:0/2899014777 <== mon.0 v1:172.21.15.46:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f47800058d0 con 0x7f47900a96b0 2024-03-25T18:08:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.291+0000 7f47877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.291+0000 7f47877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:47.292 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-25T18:08:47.292 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 10 librados: watch_flush enter 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 10 librados: watch_flush exit 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 10 client.4158.objecter successfully canceled tick 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 10 monclient: shutdown 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 >> v1:172.21.15.46:6789/0 conn(0x7f47900a96b0 legacy=0x7f47900ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 shutdown_connections 2024-03-25T18:08:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.292+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 >> 172.21.15.46:0/2899014777 conn(0x7f47900668e0 msgr2=0x7f47900b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.293+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 shutdown_connections 2024-03-25T18:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.293+0000 7f4795060640 1 -- 172.21.15.46:0/2899014777 wait complete. 2024-03-25T18:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-25T18:08:47.293+0000 7f4795060640 1 librados: shutdown 2024-03-25T18:08:47.309 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:08:47.309 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:08:48.114 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:48.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.260+0000 7f85ecc6b640 10 monclient: get_monmap_and_config 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.260+0000 7f85ecc6b640 10 monclient: build_initial_monmap 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.260+0000 7f85ecc6b640 10 monclient: monmap: 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:08:48.260991+0000 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:08:48.260991+0000 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:48.261 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.261+0000 7f85ecc6b640 1 Processor -- start 2024-03-25T18:08:48.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.261+0000 7f85ecc6b640 1 -- start start 2024-03-25T18:08:48.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.261+0000 7f85ecc6b640 10 monclient: init 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-a con 0x7f85e80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-b con 0x7f85e813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-c con 0x7f85e81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e8142520 con 0x7f85e80a98f0 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80b28a0 con 0x7f85e81430c0 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80b2a30 con 0x7f85e813f710 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:48.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.262+0000 7f85ecc6b640 10 monclient(hunting): authenticate will time out at 7237.524394s 2024-03-25T18:08:48.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.263+0000 7f85e6d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f85e81430c0 0x7f85e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36696/0 (socket says 172.21.15.46:36696) 2024-03-25T18:08:48.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.263+0000 7f85e6d9d640 1 -- 172.21.15.46:0/2824983580 learned_addr learned my addr 172.21.15.46:0/2824983580 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144588857 0 0) 0x7f85e80b2a30 con 0x7f85e813f710 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:08:48.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.572+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d0003520 con 0x7f85e813f710 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539779536 0 0) 0x7f85d0003520 con 0x7f85e813f710 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 >> v1:172.21.15.164:6789/0 conn(0x7f85e81430c0 legacy=0x7f85e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 >> v1:172.21.15.46:6789/0 conn(0x7f85e80a98f0 legacy=0x7f85e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 monclient: found mon.noname-b 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80b44c0 con 0x7f85e813f710 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: _finish_auth 0 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:48.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f85d8001670 con 0x7f85e813f710 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85ecc6b640 5 monclient: authenticate success, global_id 4178 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85ecc6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85e559a640 10 monclient: dump: 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:48.587 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:48.588 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:48.588 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.585+0000 7f85ecc6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:48.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:48.634+0000 7f9d05cc2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85d80017d0 con 0x7f85e813f710 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 1 -- 172.21.15.46:0/2824983580 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f85d8001c20 con 0x7f85e813f710 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85ecc6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85ecc6b640 4 set_mon_vals no callback set 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.642+0000 7f85e559a640 10 monclient: dump: 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:48.643 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:48.644 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.643+0000 7f85ecc6b640 10 monclient: shutdown 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.643+0000 7f85ecc6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.643+0000 7f85ecc6b640 1 -- 172.21.15.46:0/2824983580 >> v1:172.21.15.46:6790/0 conn(0x7f85e813f710 legacy=0x7f85e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.643+0000 7f85ecc6b640 1 -- 172.21.15.46:0/2824983580 shutdown_connections 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.643+0000 7f85ecc6b640 1 -- 172.21.15.46:0/2824983580 >> 172.21.15.46:0/2824983580 conn(0x7f85e80b7ea0 msgr2=0x7f85e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.644+0000 7f85ecc6b640 1 -- 172.21.15.46:0/2824983580 shutdown_connections 2024-03-25T18:08:48.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.644+0000 7f85ecc6b640 1 -- 172.21.15.46:0/2824983580 wait complete. 2024-03-25T18:08:48.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.645+0000 7f85ecc6b640 10 monclient: build_initial_monmap 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.645+0000 7f85ecc6b640 10 monclient: monmap: 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.645+0000 7f85ecc6b640 1 librados: starting msgr at 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.645+0000 7f85ecc6b640 1 librados: starting objecter 2024-03-25T18:08:48.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.645+0000 7f85ecc6b640 1 Processor -- start 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.646+0000 7f85ecc6b640 1 -- start start 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.646+0000 7f85ecc6b640 1 librados: setting wanted keys 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.646+0000 7f85ecc6b640 1 librados: calling monclient init 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.646+0000 7f85ecc6b640 10 monclient: init 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-c con 0x7f85e80a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-b con 0x7f85e813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient(hunting): picked mon.noname-a con 0x7f85e81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e81d7d80 con 0x7f85e81430c0 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e8137290 con 0x7f85e813f710 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e8137dc0 con 0x7f85e80a98f0 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:48.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.647+0000 7f85ecc6b640 10 monclient(hunting): authenticate will time out at 7237.909270s 2024-03-25T18:08:48.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85e6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f85e81430c0 0x7f85e81e2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57286/0 (socket says 172.21.15.46:57286) 2024-03-25T18:08:48.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85e6d9d640 1 -- 172.21.15.46:0/202538980 learned_addr learned my addr 172.21.15.46:0/202538980 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:48.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85d6ffd640 10 client.?.objecter ms_handle_connect 0x7f85e81430c0 2024-03-25T18:08:48.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:48.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85d6ffd640 10 client.?.objecter ms_handle_connect 0x7f85e80a98f0 2024-03-25T18:08:48.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.648+0000 7f85d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:48.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999486671 0 0) 0x7f85e8137dc0 con 0x7f85e80a98f0 2024-03-25T18:08:48.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:48.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:48.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:48.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 10 monclient(hunting): my global_id is 4181 2024-03-25T18:08:48.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.652+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8004eb0 con 0x7f85e80a98f0 2024-03-25T18:08:48.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.656+0000 7f85d6ffd640 10 client.?.objecter ms_handle_connect 0x7f85e813f710 2024-03-25T18:08:48.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.656+0000 7f85d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:48.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.659+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 648307128 0 0) 0x7f85c8004eb0 con 0x7f85e80a98f0 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85e8137dc0 con 0x7f85e80a98f0 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f85dc0019c0 con 0x7f85e80a98f0 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:48.661 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.660+0000 7f85d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 682142450 0 0) 0x7f85e8137dc0 con 0x7f85e80a98f0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 >> v1:172.21.15.164:6789/0 conn(0x7f85e813f710 legacy=0x7f85e8132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 >> v1:172.21.15.46:6789/0 conn(0x7f85e81430c0 legacy=0x7f85e81e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 1 monclient: found mon.c 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8137f50 con 0x7f85e80a98f0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.666+0000 7f85d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:48.669 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 5 monclient: authenticate success, global_id 4181 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 10 monclient: _renew_subs 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e80b72e0 con 0x7f85e80a98f0 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 10 monclient: _renew_subs 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.667+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e8133880 con 0x7f85e80a98f0 2024-03-25T18:08:48.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.668+0000 7f85ecc6b640 1 librados: init done 2024-03-25T18:08:48.671 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.669+0000 7f85ecc6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:48.671 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.669+0000 7f85ecc6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:48.671 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.669+0000 7f85ecc6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:48.671 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.669+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e80b7680 con 0x7f85e80a98f0 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85dc003670 con 0x7f85e80a98f0 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f85dc003ac0 con 0x7f85e80a98f0 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85d6ffd640 10 monclient: dump: 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:48.698 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:48.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.697+0000 7f85e4d99640 4 set_mon_vals no callback set 2024-03-25T18:08:48.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.698+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3807588493 0 0) 0x7f85dc003ce0 con 0x7f85e80a98f0 2024-03-25T18:08:48.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.699+0000 7f85e5d9b640 10 monclient: get_auth_request con 0x7f85c806db80 auth_method 0 2024-03-25T18:08:48.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.699+0000 7f85d6ffd640 10 client.4181.objecter ms_handle_connect 0x7f85c806db80 2024-03-25T18:08:48.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.702+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f85dc053630 con 0x7f85e80a98f0 2024-03-25T18:08:48.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.702+0000 7f85d6ffd640 10 client.4181.objecter ms_dispatch 0x7f85e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:48.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.702+0000 7f85d6ffd640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:48.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.702+0000 7f85d6ffd640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:48.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:48.702+0000 7f85d6ffd640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-25T18:08:48.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:08:48.728+0000 7f9d05cc2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:08:49.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.129+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85dc04f140 con 0x7f85e80a98f0 2024-03-25T18:08:49.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.129+0000 7f85d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:49.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.129+0000 7f85d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:49.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.498+0000 7f85ecc6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:49.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.498+0000 7f85ecc6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:49.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.498+0000 7f85ecc6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:08:49.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.498+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85e814f320 con 0x7f85e80a98f0 2024-03-25T18:08:49.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.573+0000 7f85d6ffd640 1 -- 172.21.15.46:0/202538980 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f85dc051070 con 0x7f85e80a98f0 2024-03-25T18:08:49.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.573+0000 7f85d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:49.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.573+0000 7f85d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:49.574 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:08:49.574 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 10 librados: watch_flush enter 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 10 librados: watch_flush exit 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 10 client.4181.objecter successfully canceled tick 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 >> v1:172.21.15.164:6800/1612084612 conn(0x7f85c806db80 legacy=0x7f85c8070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:49.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 10 monclient: shutdown 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 >> v1:172.21.15.46:6790/0 conn(0x7f85e80a98f0 legacy=0x7f85e81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 shutdown_connections 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 >> 172.21.15.46:0/202538980 conn(0x7f85e80b7ea0 msgr2=0x7f85e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.576+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 shutdown_connections 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.577+0000 7f85ecc6b640 1 -- 172.21.15.46:0/202538980 wait complete. 2024-03-25T18:08:49.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:49.577+0000 7f85ecc6b640 1 librados: shutdown 2024-03-25T18:08:49.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:53.372 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:06.814 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.397 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:06.861 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.432 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:06.874 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.469 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:06.934 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.531 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:06.995 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.591 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:07.055 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.669 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:07.118 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.676 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:07.154 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.728 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:07.177 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:53.739 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:07.216 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-25T18:08:53.994+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-25T18:08:54.016+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.088 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:07.549 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.120 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:07.580 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.127 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:07.605 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-25T18:08:54.133+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.149 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:07.610 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.180 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:07.640 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.187 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:07.666 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.291 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-03-25T18:08:54.284+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-25T18:08:54.318+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.481 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:07.922 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.482 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:07.946 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.604 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:08.068 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.718 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-03-25T18:08:54.711+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.762 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-03-25T18:08:54.755+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.783 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:08.261 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.784 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-03-25T18:08:54.775+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:08:54.790 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:08.239 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.857 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:08.298 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.861 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:08.325 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:54.967 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:08.430 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.157 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:08.606 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.172 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:08.650 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.186 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:08.647 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.217 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:08.659 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.220 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:08.685 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.237 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:08.715 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.252 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:08.711 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.276 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:08.740 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.511 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:08.960 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.520 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:08.961 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.527 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:08.991 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.537 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:09.015 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.554 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:09.015 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.556 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:09.020 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.605 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:09.084 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.639 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:09.098 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.804 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:09.253 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.816 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:09.258 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.823 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:09.287 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.824 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:09.288 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.839 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:09.317 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.900 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:09.361 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:55.959 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:09.438 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.004 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:09.464 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.100 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:09.563 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.104 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:09.553 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.119 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:09.560 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.128 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:09.592 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.132 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:09.610 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.189 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:09.650 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.249 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:09.728 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.293 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:09.753 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.368 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:09.832 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.420 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:56.421 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:09.878 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.421 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:09.845 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.421 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:09.853 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.421 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:09.884 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.482 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:09.942 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.520+0000 7efbff4a8640 10 monclient: get_monmap_and_config 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.520+0000 7efbff4a8640 10 monclient: build_initial_monmap 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.520+0000 7efbff4a8640 10 monclient: monmap: 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:08:56.522214+0000 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:08:56.522214+0000 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:08:56.522 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.521+0000 7efbff4a8640 1 Processor -- start 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.521+0000 7efbff4a8640 1 -- start start 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.521+0000 7efbff4a8640 10 monclient: init 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:56.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-c con 0x7efbf80b10d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-b con 0x7efbf80b4360 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-a con 0x7efbf80a9470 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80a9ab0 con 0x7efbf80a9470 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80b2400 con 0x7efbf80b10d0 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80aa2a0 con 0x7efbf80b4360 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbff4a8640 10 monclient(hunting): authenticate will time out at 7245.784803s 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbfd244640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efbf80b10d0 0x7efbf80b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34502/0 (socket says 172.21.15.46:34502) 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbfda45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efbf80a9470 0x7efbf814dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43168/0 (socket says 172.21.15.46:43168) 2024-03-25T18:08:56.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.522+0000 7efbfd244640 1 -- 172.21.15.46:0/3622128834 learned_addr learned my addr 172.21.15.46:0/3622128834 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519807853 0 0) 0x7efbf80b2400 con 0x7efbf80b10d0 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003520 con 0x7efbf80b10d0 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061212935 0 0) 0x7efbf80aa2a0 con 0x7efbf80b4360 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80b2400 con 0x7efbf80b4360 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286686568 0 0) 0x7efbf80a9ab0 con 0x7efbf80a9470 2024-03-25T18:08:56.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:08:56.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.525+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80aa2a0 con 0x7efbf80a9470 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 22060152 0 0) 0x7efbf80b2400 con 0x7efbf80b4360 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 >> v1:172.21.15.164:6789/0 conn(0x7efbf80b10d0 legacy=0x7efbf80b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 >> v1:172.21.15.46:6789/0 conn(0x7efbf80a9470 legacy=0x7efbf814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 monclient: found mon.noname-b 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80b14b0 con 0x7efbf80b4360 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: _finish_auth 0 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:56.530 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbe8001670 con 0x7efbf80b4360 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbff4a8640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbff4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbeffff640 10 monclient: dump: 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.531 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.532 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.532 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.528+0000 7efbff4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.529+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efbe80017d0 con 0x7efbf80b4360 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.529+0000 7efbeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 4 set_mon_vals no callback set 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbeffff640 1 -- 172.21.15.46:0/3622128834 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbe8001c20 con 0x7efbf80b4360 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbeffff640 10 monclient: dump: 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 10 monclient: shutdown 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 1 -- 172.21.15.46:0/3622128834 >> v1:172.21.15.46:6790/0 conn(0x7efbf80b4360 legacy=0x7efbf80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 1 -- 172.21.15.46:0/3622128834 shutdown_connections 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 1 -- 172.21.15.46:0/3622128834 >> 172.21.15.46:0/3622128834 conn(0x7efbf80617a0 msgr2=0x7efbf8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 1 -- 172.21.15.46:0/3622128834 shutdown_connections 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.530+0000 7efbff4a8640 1 -- 172.21.15.46:0/3622128834 wait complete. 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.531+0000 7efbff4a8640 10 monclient: build_initial_monmap 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.531+0000 7efbff4a8640 10 monclient: monmap: 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:56.537 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.531+0000 7efbff4a8640 1 librados: starting msgr at 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.531+0000 7efbff4a8640 1 librados: starting objecter 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.531+0000 7efbff4a8640 1 Processor -- start 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 -- start start 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 librados: setting wanted keys 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 librados: calling monclient init 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient: init 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-c con 0x7efbf80a9470 addr v1:172.21.15.46:6790/0 2024-03-25T18:08:56.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-b con 0x7efbf80b10d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient(hunting): picked mon.noname-a con 0x7efbf80b4360 addr v1:172.21.15.46:6789/0 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80aba40 con 0x7efbf80b4360 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80ac940 con 0x7efbf80b10d0 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80acc20 con 0x7efbf80a9470 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbff4a8640 10 monclient(hunting): authenticate will time out at 7245.795072s 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.532+0000 7efbfca43640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efbf80b10d0 0x7efbf813f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34512/0 (socket says 172.21.15.46:34512) 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbfca43640 1 -- 172.21.15.46:0/1340177408 learned_addr learned my addr 172.21.15.46:0/1340177408 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter ms_handle_connect 0x7efbf80b10d0 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter ms_handle_connect 0x7efbf80a9470 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter ms_handle_connect 0x7efbf80b4360 2024-03-25T18:08:56.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.533+0000 7efbedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:56.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280495220 0 0) 0x7efbf80acc20 con 0x7efbf80a9470 2024-03-25T18:08:56.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.535+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbdc004eb0 con 0x7efbf80a9470 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028072210 0 0) 0x7efbf80aba40 con 0x7efbf80b4360 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80acc20 con 0x7efbf80b4360 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004288583 0 0) 0x7efbf80ac940 con 0x7efbf80b10d0 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:08:56.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.536+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80aba40 con 0x7efbf80b10d0 2024-03-25T18:08:56.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034547794 0 0) 0x7efbf80aba40 con 0x7efbf80b10d0 2024-03-25T18:08:56.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf80ac940 con 0x7efbf80b10d0 2024-03-25T18:08:56.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbe8001630 con 0x7efbf80b10d0 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 10 monclient(hunting): dump: 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.543 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.544 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.541+0000 7efbedffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:56.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957240543 0 0) 0x7efbdc004eb0 con 0x7efbf80a9470 2024-03-25T18:08:56.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbf80aba40 con 0x7efbf80a9470 2024-03-25T18:08:56.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbf40012f0 con 0x7efbf80a9470 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): dump: 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451311782 0 0) 0x7efbf80acc20 con 0x7efbf80b4360 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efbdc004eb0 con 0x7efbf80b4360 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbf0001bf0 con 0x7efbf80b4360 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.542+0000 7efbedffb640 10 monclient(hunting): dump: 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.546 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.547 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.547 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:10.023 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1003698213 0 0) 0x7efbf80ac940 con 0x7efbf80b10d0 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 >> v1:172.21.15.46:6790/0 conn(0x7efbf80a9470 legacy=0x7efbf80ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 >> v1:172.21.15.46:6789/0 conn(0x7efbf80b4360 legacy=0x7efbf81e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 1 monclient: found mon.b 2024-03-25T18:08:56.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80acf00 con 0x7efbf80b10d0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 10 monclient: _renew_subs 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf813eb50 con 0x7efbf80b10d0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 10 monclient: _renew_subs 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf813d770 con 0x7efbf80b10d0 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.544+0000 7efbff4a8640 1 librados: init done 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.545+0000 7efbff4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:56.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.545+0000 7efbff4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:56.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.545+0000 7efbff4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:56.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.545+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf80627b0 con 0x7efbf80b10d0 2024-03-25T18:08:56.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efbe80038a0 con 0x7efbf80b10d0 2024-03-25T18:08:56.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:56.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efbe8003c30 con 0x7efbf80b10d0 2024-03-25T18:08:56.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 10 monclient: dump: 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:08:56.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbef7fe640 4 set_mon_vals no callback set 2024-03-25T18:08:56.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.559+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7efbe8003e50 con 0x7efbf80b10d0 2024-03-25T18:08:56.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.560+0000 7efbfd244640 10 monclient: get_auth_request con 0x7efbdc06e110 auth_method 0 2024-03-25T18:08:56.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.560+0000 7efbedffb640 10 client.4207.objecter ms_handle_connect 0x7efbdc06e110 2024-03-25T18:08:56.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.561+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7efbe8001d20 con 0x7efbf80b10d0 2024-03-25T18:08:56.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.561+0000 7efbedffb640 10 client.4207.objecter ms_dispatch 0x7efbf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:56.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.561+0000 7efbedffb640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:56.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.561+0000 7efbedffb640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:56.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.561+0000 7efbedffb640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T18:08:56.591 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:10.051 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.637 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:10.101 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.687 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:10.136 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.689 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:10.167 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.716 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:10.157 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.720 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:10.184 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.785 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:10.246 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.791+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efbe8050280 con 0x7efbf80b10d0 2024-03-25T18:08:56.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.791+0000 7efbedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:56.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.791+0000 7efbedffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:56.813 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:10.292 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.880 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:10.340 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.901 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:10.365 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.974+0000 7efbff4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:56.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.974+0000 7efbff4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:56.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.974+0000 7efbff4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:08:56.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:56.974+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efbf813ece0 con 0x7efbf80b10d0 2024-03-25T18:08:56.978 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:10.456 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.989 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:10.438 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:56.999 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:10.463 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.015 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:10.457 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.020+0000 7efbedffb640 1 -- 172.21.15.46:0/1340177408 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7efbf813ece0 con 0x7efbf80b10d0 2024-03-25T18:08:57.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.020+0000 7efbedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:57.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.020+0000 7efbedffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:57.021 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:08:57.021 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 10 librados: watch_flush enter 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 10 librados: watch_flush exit 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 10 client.4207.objecter successfully canceled tick 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 >> v1:172.21.15.164:6800/1612084612 conn(0x7efbdc06e110 legacy=0x7efbdc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 10 monclient: shutdown 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 >> v1:172.21.15.164:6789/0 conn(0x7efbf80b10d0 legacy=0x7efbf813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 shutdown_connections 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 >> 172.21.15.46:0/1340177408 conn(0x7efbf80617a0 msgr2=0x7efbf8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:57.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 shutdown_connections 2024-03-25T18:08:57.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 -- 172.21.15.46:0/1340177408 wait complete. 2024-03-25T18:08:57.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:08:57.022+0000 7efbff4a8640 1 librados: shutdown 2024-03-25T18:08:57.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:57.074 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:10.535 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.106 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:10.585 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.123 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:10.584 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.186 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:10.647 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.193 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:10.653 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.271 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:10.749 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.277 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:10.726 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.372 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:10.836 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.398 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:10.877 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.426 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:10.890 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.487 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:10.947 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.552 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:11.017 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.588 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:11.030 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.627 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:11.092 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.646 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:11.088 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.687 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:11.166 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.730 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:11.191 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.779 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:11.239 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.788 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:11.249 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.826 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:11.304 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.837 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:11.286 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.882 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:11.360 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:57.891 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:11.340 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:58.211 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:11.689 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:58.267 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:11.745 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:58.318 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:11.778 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:58.396 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:11.856 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:02.824 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:16.288 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:02.876 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:16.318 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:02.923 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:16.388 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:03.008 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:16.457 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:03.565 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:17.026 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:03.772 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:17.249 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:03.840 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.979+0000 7f8c84d9f640 10 monclient: get_monmap_and_config 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.979+0000 7f8c84d9f640 10 monclient: build_initial_monmap 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.979+0000 7f8c84d9f640 10 monclient: monmap: 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:03.981448+0000 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:03.981448+0000 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:03.981 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:03.982 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:03.982 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:03.982 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:03.982 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.980+0000 7f8c84d9f640 1 Processor -- start 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.980+0000 7f8c84d9f640 1 -- start start 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.980+0000 7f8c84d9f640 10 monclient: init 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:03.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c800b49a0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c800a9680 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c800b3fa0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.981+0000 7f8c84d9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8013fe70 con 0x7f8c800a9680 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c84d9f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800ab250 con 0x7f8c800b49a0 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c84d9f640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800a9b10 con 0x7f8c800b3fa0 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c84d9f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c84d9f640 10 monclient(hunting): authenticate will time out at 7253.244733s 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c7effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c800a9680 0x7f8c80145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49080/0 (socket says 172.21.15.46:49080) 2024-03-25T18:09:03.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.982+0000 7f8c7effd640 1 -- 172.21.15.46:0/2812737878 learned_addr learned my addr 172.21.15.46:0/2812737878 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004797526 0 0) 0x7f8c800a9b10 con 0x7f8c800b3fa0 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:09:03.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c68003520 con 0x7f8c800b3fa0 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990434702 0 0) 0x7f8c800ab250 con 0x7f8c800b49a0 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c800a9b10 con 0x7f8c800b49a0 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186179852 0 0) 0x7f8c74001110 con 0x7f8c800a9680 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:09:03.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.986+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c800ab250 con 0x7f8c800a9680 2024-03-25T18:09:03.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1718904069 0 0) 0x7f8c800ab250 con 0x7f8c800a9680 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 >> v1:172.21.15.46:6790/0 conn(0x7f8c800b3fa0 legacy=0x7f8c80145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 >> v1:172.21.15.164:6789/0 conn(0x7f8c800b49a0 legacy=0x7f8c8013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80151b50 con 0x7f8c800a9680 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c740015a0 con 0x7f8c800a9680 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:03.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c7e7fc640 10 monclient: dump: 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c84d9f640 5 monclient: authenticate success, global_id 4206 2024-03-25T18:09:03.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.991+0000 7f8c84d9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:03.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.993+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c74001700 con 0x7f8c800a9680 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.993+0000 7f8c7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.993+0000 7f8c84d9f640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.993+0000 7f8c84d9f640 4 set_mon_vals no callback set 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c7e7fc640 1 -- 172.21.15.46:0/2812737878 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c74001b50 con 0x7f8c800a9680 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c7e7fc640 10 monclient: dump: 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:03.996 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 10 monclient: shutdown 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 1 -- 172.21.15.46:0/2812737878 >> v1:172.21.15.46:6789/0 conn(0x7f8c800a9680 legacy=0x7f8c80145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 1 -- 172.21.15.46:0/2812737878 shutdown_connections 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 1 -- 172.21.15.46:0/2812737878 >> 172.21.15.46:0/2812737878 conn(0x7f8c800615c0 msgr2=0x7f8c800619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 1 -- 172.21.15.46:0/2812737878 shutdown_connections 2024-03-25T18:09:03.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.994+0000 7f8c84d9f640 1 -- 172.21.15.46:0/2812737878 wait complete. 2024-03-25T18:09:03.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 10 monclient: build_initial_monmap 2024-03-25T18:09:03.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 10 monclient: monmap: 2024-03-25T18:09:03.999 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:03.999 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 librados: starting msgr at 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 librados: starting objecter 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 Processor -- start 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 -- start start 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 librados: setting wanted keys 2024-03-25T18:09:04.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 1 librados: calling monclient init 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.995+0000 7f8c84d9f640 10 monclient: init 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c800b3fa0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c800b49a0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c80150fe0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c801433e0 con 0x7f8c800b3fa0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80143870 con 0x7f8c800b49a0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c80144430 con 0x7f8c80150fe0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.996+0000 7f8c84d9f640 10 monclient(hunting): authenticate will time out at 7253.259266s 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c7f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c800b3fa0 0x7f8c8013e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49082/0 (socket says 172.21.15.46:49082) 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c7f7fe640 1 -- 172.21.15.46:0/48816880 learned_addr learned my addr 172.21.15.46:0/48816880 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter ms_handle_connect 0x7f8c800b3fa0 2024-03-25T18:09:04.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:04.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter ms_handle_connect 0x7f8c800b49a0 2024-03-25T18:09:04.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:04.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter ms_handle_connect 0x7f8c80150fe0 2024-03-25T18:09:04.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:03.997+0000 7f8c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621581190 0 0) 0x7f8c801433e0 con 0x7f8c800b3fa0 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c004eb0 con 0x7f8c800b3fa0 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946090530 0 0) 0x7f8c80143870 con 0x7f8c800b49a0 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:04.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c801433e0 con 0x7f8c800b49a0 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728034071 0 0) 0x7f8c80144430 con 0x7f8c80150fe0 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:09:04.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.000+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80143870 con 0x7f8c80150fe0 2024-03-25T18:09:04.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532444141 0 0) 0x7f8c801433e0 con 0x7f8c800b49a0 2024-03-25T18:09:04.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c80144430 con 0x7f8c800b49a0 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c74001a10 con 0x7f8c800b49a0 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:04.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.006+0000 7f8c67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729692643 0 0) 0x7f8c80143870 con 0x7f8c80150fe0 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c801433e0 con 0x7f8c80150fe0 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c78001c90 con 0x7f8c80150fe0 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.010 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3888462402 0 0) 0x7f8c6c004eb0 con 0x7f8c800b3fa0 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.007+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c80143870 con 0x7f8c800b3fa0 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.008+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c70001990 con 0x7f8c800b3fa0 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.008+0000 7f8c67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.008+0000 7f8c67fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.008+0000 7f8c67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.011 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:04.012 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:04.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1144314050 0 0) 0x7f8c80144430 con 0x7f8c800b49a0 2024-03-25T18:09:04.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 >> v1:172.21.15.46:6790/0 conn(0x7f8c80150fe0 legacy=0x7f8c801e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:04.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 >> v1:172.21.15.46:6789/0 conn(0x7f8c800b3fa0 legacy=0x7f8c8013e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:04.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 1 monclient: found mon.b 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80144710 con 0x7f8c800b49a0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 5 monclient: authenticate success, global_id 4213 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 10 monclient: _renew_subs 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8014f350 con 0x7f8c800b49a0 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 10 monclient: _renew_subs 2024-03-25T18:09:04.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c80150560 con 0x7f8c800b49a0 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.010+0000 7f8c84d9f640 1 librados: init done 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.011+0000 7f8c84d9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.011+0000 7f8c84d9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.012+0000 7f8c84d9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:04.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.012+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c80144c30 con 0x7f8c800b49a0 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c740013a0 con 0x7f8c800b49a0 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c74001b90 con 0x7f8c800b49a0 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c67fff640 10 monclient: dump: 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:04.032 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:04.033 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.030+0000 7f8c7dffb640 4 set_mon_vals no callback set 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.031+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f8c7401a940 con 0x7f8c800b49a0 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.032+0000 7f8c7f7fe640 10 monclient: get_auth_request con 0x7f8c6c06e0f0 auth_method 0 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.032+0000 7f8c67fff640 10 client.4213.objecter ms_handle_connect 0x7f8c6c06e0f0 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.033+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f8c74054660 con 0x7f8c800b49a0 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.033+0000 7f8c67fff640 10 client.4213.objecter ms_dispatch 0x7f8c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.033+0000 7f8c67fff640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.033+0000 7f8c67fff640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:04.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.033+0000 7f8c67fff640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T18:09:04.402 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.400+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c7404ffa0 con 0x7f8c800b49a0 2024-03-25T18:09:04.402 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.400+0000 7f8c67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:04.402 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.400+0000 7f8c67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:04.435 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:17.894 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:04.498 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:17.977 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:04.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.620+0000 7f8c84d9f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:04.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.620+0000 7f8c84d9f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:04.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.620+0000 7f8c84d9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:04.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.620+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c80143590 con 0x7f8c800b49a0 2024-03-25T18:09:04.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.646+0000 7f8c67fff640 1 -- 172.21.15.46:0/48816880 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f8c80143590 con 0x7f8c800b49a0 2024-03-25T18:09:04.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.646+0000 7f8c67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:04.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.646+0000 7f8c67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:04.648 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:04.648 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.648+0000 7f8c84d9f640 10 librados: watch_flush enter 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.648+0000 7f8c84d9f640 10 librados: watch_flush exit 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.648+0000 7f8c84d9f640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.648+0000 7f8c84d9f640 10 client.4213.objecter successfully canceled tick 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 >> v1:172.21.15.164:6800/1612084612 conn(0x7f8c6c06e0f0 legacy=0x7f8c6c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 10 monclient: shutdown 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 >> v1:172.21.15.164:6789/0 conn(0x7f8c800b49a0 legacy=0x7f8c8014f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 shutdown_connections 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 >> 172.21.15.46:0/48816880 conn(0x7f8c800615c0 msgr2=0x7f8c800b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:04.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 shutdown_connections 2024-03-25T18:09:04.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 -- 172.21.15.46:0/48816880 wait complete. 2024-03-25T18:09:04.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:04.649+0000 7f8c84d9f640 1 librados: shutdown 2024-03-25T18:09:04.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:08.387 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:21.850 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:08.394 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:21.857 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:08.730 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:22.172 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:08.734 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:22.176 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:08.778 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:22.242 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:08.784 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:22.248 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.159 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:22.619 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.164 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:22.625 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.231 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:22.708 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.237 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:22.715 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.323 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:22.772 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:09.327 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:22.777 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:10.151 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:23.610 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:10.155 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:23.614 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:10.394 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:23.872 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:10.398 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:23.876 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:11.469 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:11.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.557+0000 7fa746fd6640 10 monclient: get_monmap_and_config 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.557+0000 7fa746fd6640 10 monclient: build_initial_monmap 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.557+0000 7fa746fd6640 10 monclient: monmap: 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:11.559177+0000 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:11.559177+0000 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:11.559 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 1 Processor -- start 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 1 -- start start 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient: init 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fa7400b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fa7400b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fa74014be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.558+0000 7fa746fd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400af940 con 0x7fa7400b4d60 2024-03-25T18:09:11.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa746fd6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400b0a70 con 0x7fa7400b1730 2024-03-25T18:09:11.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa746fd6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400b0d50 con 0x7fa74014be90 2024-03-25T18:09:11.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa746fd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:11.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa746fd6640 10 monclient(hunting): authenticate will time out at 7260.821629s 2024-03-25T18:09:11.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa73ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa7400b1730 0x7fa7400b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45330/0 (socket says 172.21.15.46:45330) 2024-03-25T18:09:11.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.559+0000 7fa73ffff640 1 -- 172.21.15.46:0/352707183 learned_addr learned my addr 172.21.15.46:0/352707183 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:11.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132837800 0 0) 0x7fa7400b0a70 con 0x7fa7400b1730 2024-03-25T18:09:11.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.562+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa730003520 con 0x7fa7400b1730 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611466373 0 0) 0x7fa7400af940 con 0x7fa7400b4d60 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:09:11.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.563+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7400b0a70 con 0x7fa7400b4d60 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805078730 0 0) 0x7fa7400b0d50 con 0x7fa74014be90 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:09:11.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.564+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7400af940 con 0x7fa74014be90 2024-03-25T18:09:11.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3172512082 0 0) 0x7fa730003520 con 0x7fa7400b1730 2024-03-25T18:09:11.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 >> v1:172.21.15.46:6790/0 conn(0x7fa74014be90 legacy=0x7fa74014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 >> v1:172.21.15.46:6789/0 conn(0x7fa7400b4d60 legacy=0x7fa74013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400b1030 con 0x7fa7400b1730 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa72c001690 con 0x7fa7400b1730 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa73f7fe640 10 monclient: dump: 2024-03-25T18:09:11.570 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa746fd6640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:09:11.571 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.567+0000 7fa746fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.569+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa72c0018e0 con 0x7fa7400b1730 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.569+0000 7fa73f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.569+0000 7fa746fd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.569+0000 7fa746fd6640 4 set_mon_vals no callback set 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 10 monclient: shutdown 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa73f7fe640 1 -- 172.21.15.46:0/352707183 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa72c001c70 con 0x7fa7400b1730 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa73f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa73f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa73f7fe640 10 monclient: dump: 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.573 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 1 -- 172.21.15.46:0/352707183 >> v1:172.21.15.164:6789/0 conn(0x7fa7400b1730 legacy=0x7fa7400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 1 -- 172.21.15.46:0/352707183 shutdown_connections 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 1 -- 172.21.15.46:0/352707183 >> 172.21.15.46:0/352707183 conn(0x7fa7400617a0 msgr2=0x7fa740061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 1 -- 172.21.15.46:0/352707183 shutdown_connections 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.570+0000 7fa746fd6640 1 -- 172.21.15.46:0/352707183 wait complete. 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.571+0000 7fa746fd6640 10 monclient: build_initial_monmap 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.571+0000 7fa746fd6640 10 monclient: monmap: 2024-03-25T18:09:11.574 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:11.575 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 librados: starting msgr at 2024-03-25T18:09:11.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 librados: starting objecter 2024-03-25T18:09:11.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 Processor -- start 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 -- start start 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 librados: setting wanted keys 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 1 librados: calling monclient init 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.572+0000 7fa746fd6640 10 monclient: init 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.573+0000 7fa746fd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.573+0000 7fa746fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.573+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fa7400b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.573+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fa7400b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fa74014be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74013b360 con 0x7fa7400b1730 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa740141e60 con 0x7fa7400b4d60 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa740142760 con 0x7fa74014be90 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:11.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa746fd6640 10 monclient(hunting): authenticate will time out at 7260.836956s 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa7400b4d60 0x7fa740141750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45336/0 (socket says 172.21.15.46:45336) 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73ffff640 1 -- 172.21.15.46:0/2399996839 learned_addr learned my addr 172.21.15.46:0/2399996839 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73d7fa640 10 client.?.objecter ms_handle_connect 0x7fa7400b4d60 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73d7fa640 10 client.?.objecter ms_handle_connect 0x7fa7400b1730 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.574+0000 7fa73d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.575+0000 7fa73d7fa640 10 client.?.objecter ms_handle_connect 0x7fa74014be90 2024-03-25T18:09:11.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.575+0000 7fa73d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743410365 0 0) 0x7fa740141e60 con 0x7fa7400b4d60 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa724004eb0 con 0x7fa7400b4d60 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765155222 0 0) 0x7fa740142760 con 0x7fa74014be90 2024-03-25T18:09:11.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740141e60 con 0x7fa74014be90 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694596941 0 0) 0x7fa74013b360 con 0x7fa7400b1730 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:09:11.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.577+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740142760 con 0x7fa7400b1730 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031737490 0 0) 0x7fa740142760 con 0x7fa7400b1730 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa74013b360 con 0x7fa7400b1730 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa734001870 con 0x7fa7400b1730 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): dump: 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.587 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1514101571 0 0) 0x7fa724004eb0 con 0x7fa7400b4d60 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa740142760 con 0x7fa7400b4d60 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa72c001bf0 con 0x7fa7400b4d60 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.584+0000 7fa73d7fa640 10 monclient(hunting): dump: 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.588 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604384634 0 0) 0x7fa740141e60 con 0x7fa74014be90 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa724004eb0 con 0x7fa74014be90 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa738001b50 con 0x7fa74014be90 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.585+0000 7fa73d7fa640 10 monclient(hunting): dump: 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.589 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4184568702 0 0) 0x7fa740142760 con 0x7fa7400b4d60 2024-03-25T18:09:11.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 >> v1:172.21.15.46:6790/0 conn(0x7fa74014be90 legacy=0x7fa7401e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 >> v1:172.21.15.46:6789/0 conn(0x7fa7400b1730 legacy=0x7fa74013ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 1 monclient: found mon.b 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740138f60 con 0x7fa7400b4d60 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa73d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:09:11.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 10 monclient: _renew_subs 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7401390f0 con 0x7fa7400b4d60 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 10 monclient: _renew_subs 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa740143140 con 0x7fa7400b4d60 2024-03-25T18:09:11.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.589+0000 7fa746fd6640 1 librados: init done 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa746fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa746fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa746fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400627b0 con 0x7fa7400b4d60 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa72c004bf0 con 0x7fa7400b4d60 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa73d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:11.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.591+0000 7fa73effd640 4 set_mon_vals no callback set 2024-03-25T18:09:11.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.671+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa72c004f00 con 0x7fa7400b4d60 2024-03-25T18:09:11.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.671+0000 7fa73d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.671+0000 7fa73d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.671+0000 7fa73d7fa640 10 monclient: dump: 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.673 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:11.674 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:11.675 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.672+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fa72c01aed0 con 0x7fa7400b4d60 2024-03-25T18:09:11.676 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.673+0000 7fa744d72640 10 monclient: get_auth_request con 0x7fa724076940 auth_method 0 2024-03-25T18:09:11.676 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.673+0000 7fa73d7fa640 10 client.4219.objecter ms_handle_connect 0x7fa724076940 2024-03-25T18:09:11.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.803+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fa72c054980 con 0x7fa7400b4d60 2024-03-25T18:09:11.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.803+0000 7fa73d7fa640 10 client.4219.objecter ms_dispatch 0x7fa740000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:11.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.803+0000 7fa73d7fa640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:11.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.803+0000 7fa73d7fa640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:11.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.803+0000 7fa73d7fa640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T18:09:11.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.937+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa72c050490 con 0x7fa7400b4d60 2024-03-25T18:09:11.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.938+0000 7fa73d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:11.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:11.938+0000 7fa73d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:12.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.172+0000 7fa746fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:12.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.172+0000 7fa746fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:12.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.172+0000 7fa746fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:12.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.172+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7401395d0 con 0x7fa7400b4d60 2024-03-25T18:09:12.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.198+0000 7fa73d7fa640 1 -- 172.21.15.46:0/2399996839 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fa72c0523c0 con 0x7fa7400b4d60 2024-03-25T18:09:12.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.198+0000 7fa73d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:12.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.198+0000 7fa73d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:12.200 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:12.200 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 10 librados: watch_flush enter 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 10 librados: watch_flush exit 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 10 client.4219.objecter successfully canceled tick 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 >> v1:172.21.15.164:6800/1612084612 conn(0x7fa724076940 legacy=0x7fa724078e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 10 monclient: shutdown 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:12.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.200+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 >> v1:172.21.15.164:6789/0 conn(0x7fa7400b4d60 legacy=0x7fa740141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:12.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.201+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 shutdown_connections 2024-03-25T18:09:12.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.201+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 >> 172.21.15.46:0/2399996839 conn(0x7fa7400617a0 msgr2=0x7fa740137430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:12.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.201+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 shutdown_connections 2024-03-25T18:09:12.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.201+0000 7fa746fd6640 1 -- 172.21.15.46:0/2399996839 wait complete. 2024-03-25T18:09:12.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:12.201+0000 7fa746fd6640 1 librados: shutdown 2024-03-25T18:09:12.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:12.731 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:00:26.194 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:12.790 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:00:26.232 35503== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:13.217 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:00:26.681 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:13.590 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:00:27.050 35796== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:13.602 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:00:27.051 35502== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:13.809 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:00:27.286 35794== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:14.951 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:00:28.410 35797== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:15.126 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:00:28.604 35792== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:19.026 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.115+0000 7fc91a196640 10 monclient: get_monmap_and_config 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.115+0000 7fc91a196640 10 monclient: build_initial_monmap 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.115+0000 7fc91a196640 10 monclient: monmap: 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:19.117608+0000 2024-03-25T18:09:19.117 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:19.117608+0000 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:19.118 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.116+0000 7fc91a196640 1 Processor -- start 2024-03-25T18:09:19.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.116+0000 7fc91a196640 1 -- start start 2024-03-25T18:09:19.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.116+0000 7fc91a196640 10 monclient: init 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-c con 0x7fc9140b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-b con 0x7fc9140b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-a con 0x7fc91414be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140af940 con 0x7fc91414be90 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140b0a70 con 0x7fc9140b4d60 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140b0d50 con 0x7fc9140b1730 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.117+0000 7fc91a196640 10 monclient(hunting): authenticate will time out at 7268.380417s 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.118+0000 7fc913fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc91414be90 0x7fc91414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54350/0 (socket says 172.21.15.46:54350) 2024-03-25T18:09:19.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.118+0000 7fc913fff640 1 -- 172.21.15.46:0/2340981629 learned_addr learned my addr 172.21.15.46:0/2340981629 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327392613 0 0) 0x7fc9140b0a70 con 0x7fc9140b4d60 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8fc003520 con 0x7fc9140b4d60 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.121+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280768343 0 0) 0x7fc9140b0d50 con 0x7fc9140b1730 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:19.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9140b0a70 con 0x7fc9140b1730 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118496697 0 0) 0x7fc9140af940 con 0x7fc91414be90 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:09:19.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.122+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9140b0d50 con 0x7fc91414be90 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416558569 0 0) 0x7fc9140b0d50 con 0x7fc91414be90 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 >> v1:172.21.15.46:6790/0 conn(0x7fc9140b1730 legacy=0x7fc9140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 >> v1:172.21.15.164:6789/0 conn(0x7fc9140b4d60 legacy=0x7fc91413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 monclient: found mon.noname-a 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140b1030 con 0x7fc91414be90 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:19.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc90c001930 con 0x7fc91414be90 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc9127fc640 10 monclient: dump: 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc91a196640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:09:19.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.125+0000 7fc91a196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:19.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc90c001b50 con 0x7fc91414be90 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 1 -- 172.21.15.46:0/2340981629 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc90c001ee0 con 0x7fc91414be90 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc9127fc640 10 monclient: dump: 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:19.132 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc91a196640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc91a196640 4 set_mon_vals no callback set 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc91a196640 10 monclient: shutdown 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc91a196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.128+0000 7fc91a196640 1 -- 172.21.15.46:0/2340981629 >> v1:172.21.15.46:6789/0 conn(0x7fc91414be90 legacy=0x7fc91414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 1 -- 172.21.15.46:0/2340981629 shutdown_connections 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 1 -- 172.21.15.46:0/2340981629 >> 172.21.15.46:0/2340981629 conn(0x7fc9140617a0 msgr2=0x7fc914061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 1 -- 172.21.15.46:0/2340981629 shutdown_connections 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 1 -- 172.21.15.46:0/2340981629 wait complete. 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 10 monclient: build_initial_monmap 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.129+0000 7fc91a196640 10 monclient: monmap: 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:19.133 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 librados: starting msgr at 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 librados: starting objecter 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 Processor -- start 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 -- start start 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 librados: setting wanted keys 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 1 librados: calling monclient init 2024-03-25T18:09:19.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.130+0000 7fc91a196640 10 monclient: init 2024-03-25T18:09:19.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-c con 0x7fc9140b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-b con 0x7fc9140b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient(hunting): picked mon.noname-a con 0x7fc91414be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914141dc0 con 0x7fc91414be90 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914140380 con 0x7fc9140b4d60 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914140660 con 0x7fc9140b1730 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.131+0000 7fc91a196640 10 monclient(hunting): authenticate will time out at 7268.394249s 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc912ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc9140b4d60 0x7fc9141d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:60778/0 (socket says 172.21.15.46:60778) 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc912ffd640 1 -- 172.21.15.46:0/976095823 learned_addr learned my addr 172.21.15.46:0/976095823 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc8fbfff640 10 client.?.objecter ms_handle_connect 0x7fc9140b4d60 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc8fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc8fbfff640 10 client.?.objecter ms_handle_connect 0x7fc91414be90 2024-03-25T18:09:19.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.132+0000 7fc8fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366154062 0 0) 0x7fc914141dc0 con 0x7fc91414be90 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900004eb0 con 0x7fc91414be90 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250755713 0 0) 0x7fc914140380 con 0x7fc9140b4d60 2024-03-25T18:09:19.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:19.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:19.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:19.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:09:19.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.135+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914141dc0 con 0x7fc9140b4d60 2024-03-25T18:09:19.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.139+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1911185234 0 0) 0x7fc900004eb0 con 0x7fc91414be90 2024-03-25T18:09:19.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc914140380 con 0x7fc91414be90 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc90c001e60 con 0x7fc91414be90 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 10 monclient(hunting): dump: 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.140+0000 7fc8fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.142+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807704677 0 0) 0x7fc914141dc0 con 0x7fc9140b4d60 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900004eb0 con 0x7fc9140b4d60 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc9080019f0 con 0x7fc9140b4d60 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient(hunting): dump: 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:19.144 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.145 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:19.145 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:19.145 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:19.145 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.145 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2089613111 0 0) 0x7fc914140380 con 0x7fc91414be90 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 >> v1:172.21.15.46:6790/0 conn(0x7fc9140b1730 legacy=0x7fc914131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 >> v1:172.21.15.164:6789/0 conn(0x7fc9140b4d60 legacy=0x7fc9141d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 monclient: found mon.a 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9141326e0 con 0x7fc91414be90 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc8fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 10 monclient: _renew_subs 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc914140b80 con 0x7fc91414be90 2024-03-25T18:09:19.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 10 monclient: _renew_subs 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc914137c80 con 0x7fc91414be90 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.143+0000 7fc91a196640 1 librados: init done 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.145+0000 7fc91a196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.145+0000 7fc91a196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.145+0000 7fc91a196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.145+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9141407f0 con 0x7fc91414be90 2024-03-25T18:09:19.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc90c001fc0 con 0x7fc91414be90 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc90c004fb0 con 0x7fc91414be90 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc8fbfff640 10 monclient: dump: 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:19.280 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:19.281 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:19.281 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:19.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.278+0000 7fc911ffb640 4 set_mon_vals no callback set 2024-03-25T18:09:19.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.280+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fc90c01af80 con 0x7fc91414be90 2024-03-25T18:09:19.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.281+0000 7fc9137fe640 10 monclient: get_auth_request con 0x7fc90006df60 auth_method 0 2024-03-25T18:09:19.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.281+0000 7fc8fbfff640 10 client.4221.objecter ms_handle_connect 0x7fc90006df60 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fc90c054950 con 0x7fc91414be90 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 10 client.4221.objecter ms_dispatch 0x7fc914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc90c050460 con 0x7fc91414be90 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:19.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.435+0000 7fc8fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:19.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.705+0000 7fc91a196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:19.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.705+0000 7fc91a196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:19.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.705+0000 7fc91a196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:09:19.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.705+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc914137f70 con 0x7fc91414be90 2024-03-25T18:09:19.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.732+0000 7fc8fbfff640 1 -- 172.21.15.46:0/976095823 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fc90c052390 con 0x7fc91414be90 2024-03-25T18:09:19.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.732+0000 7fc8fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:19.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.732+0000 7fc8fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:19.733 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:19.734 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:19.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.734+0000 7fc91a196640 10 librados: watch_flush enter 2024-03-25T18:09:19.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.734+0000 7fc91a196640 10 librados: watch_flush exit 2024-03-25T18:09:19.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.734+0000 7fc91a196640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T18:09:19.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.734+0000 7fc91a196640 10 client.4221.objecter successfully canceled tick 2024-03-25T18:09:19.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.734+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 >> v1:172.21.15.164:6800/1612084612 conn(0x7fc90006df60 legacy=0x7fc900070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 10 monclient: shutdown 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 >> v1:172.21.15.46:6789/0 conn(0x7fc91414be90 legacy=0x7fc9141e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 shutdown_connections 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 >> 172.21.15.46:0/976095823 conn(0x7fc9140617a0 msgr2=0x7fc914061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 shutdown_connections 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 -- 172.21.15.46:0/976095823 wait complete. 2024-03-25T18:09:19.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:19.735+0000 7fc91a196640 1 librados: shutdown 2024-03-25T18:09:19.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:20.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-25T18:09:20.957+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:09:21.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-25T18:09:21.126+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:09:21.561 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-25T18:09:21.557+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:09:21.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-25T18:09:21.946+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:09:22.372 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-03-25T18:09:22.368+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:09:22.898 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-03-25T18:09:22.894+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:09:24.630 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-03-25T18:09:24.625+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:09:25.374 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-03-25T18:09:25.370+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:09:26.557 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:26.671 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.671+0000 7f244780e640 10 monclient: get_monmap_and_config 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.671+0000 7f244780e640 10 monclient: build_initial_monmap 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.671+0000 7f244780e640 10 monclient: monmap: 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:26.672005+0000 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:26.672005+0000 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:26.672 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:26.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.671+0000 7f244780e640 1 Processor -- start 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 1 -- start start 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient: init 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient(hunting): picked mon.noname-c con 0x7f24400a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient(hunting): picked mon.noname-a con 0x7f244013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient(hunting): picked mon.noname-b con 0x7f24401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440142520 con 0x7f244013f710 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400b28a0 con 0x7f24400a98f0 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400b2a30 con 0x7f24401430c0 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:26.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.672+0000 7f244780e640 10 monclient(hunting): authenticate will time out at 7275.934386s 2024-03-25T18:09:26.710 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.709+0000 7f24455aa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f24400a98f0 0x7f24400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47728/0 (socket says 172.21.15.46:47728) 2024-03-25T18:09:26.710 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.709+0000 7f24455aa640 1 -- 172.21.15.46:0/4224842265 learned_addr learned my addr 172.21.15.46:0/4224842265 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462613196 0 0) 0x7f2440142520 con 0x7f244013f710 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:09:26.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.728+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242c003520 con 0x7f244013f710 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794354946 0 0) 0x7f24400b28a0 con 0x7f24400a98f0 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:09:26.741 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.741+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2440142520 con 0x7f24400a98f0 2024-03-25T18:09:26.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635069633 0 0) 0x7f24400b2a30 con 0x7f24401430c0 2024-03-25T18:09:26.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24400b28a0 con 0x7f24401430c0 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983966785 0 0) 0x7f242c003520 con 0x7f244013f710 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 >> v1:172.21.15.46:6790/0 conn(0x7f24401430c0 legacy=0x7f244014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 >> v1:172.21.15.164:6789/0 conn(0x7f24400a98f0 legacy=0x7f24400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 monclient: found mon.noname-a 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400b44c0 con 0x7f244013f710 2024-03-25T18:09:26.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.748+0000 7f2437fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2430001690 con 0x7f244013f710 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f2437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f2437fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f2437fff640 10 monclient: dump: 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:26.751 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:26.752 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:26.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f244780e640 5 monclient: authenticate success, global_id 4248 2024-03-25T18:09:26.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.749+0000 7f244780e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2430001870 con 0x7f244013f710 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 1 -- 172.21.15.46:0/4224842265 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2430001c00 con 0x7f244013f710 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f244780e640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f244780e640 4 set_mon_vals no callback set 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:26.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.776+0000 7f2437fff640 10 monclient: dump: 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:26.778 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 10 monclient: shutdown 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 1 -- 172.21.15.46:0/4224842265 >> v1:172.21.15.46:6789/0 conn(0x7f244013f710 legacy=0x7f24401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 1 -- 172.21.15.46:0/4224842265 shutdown_connections 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 1 -- 172.21.15.46:0/4224842265 >> 172.21.15.46:0/4224842265 conn(0x7f24400b7ea0 msgr2=0x7f24400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:26.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 1 -- 172.21.15.46:0/4224842265 shutdown_connections 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.777+0000 7f244780e640 1 -- 172.21.15.46:0/4224842265 wait complete. 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.778+0000 7f244780e640 10 monclient: build_initial_monmap 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.778+0000 7f244780e640 10 monclient: monmap: 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 librados: starting msgr at 2024-03-25T18:09:26.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 librados: starting objecter 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 Processor -- start 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 -- start start 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 librados: setting wanted keys 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 1 librados: calling monclient init 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.779+0000 7f244780e640 10 monclient: init 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient(hunting): picked mon.noname-b con 0x7f24400a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient(hunting): picked mon.noname-c con 0x7f24401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient(hunting): picked mon.noname-a con 0x7f24401369e0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440137480 con 0x7f24401369e0 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244013e9e0 con 0x7f24400a98f0 2024-03-25T18:09:26.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244013eb70 con 0x7f24401430c0 2024-03-25T18:09:26.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:26.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.780+0000 7f244780e640 10 monclient(hunting): authenticate will time out at 7276.041801s 2024-03-25T18:09:26.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.788+0000 7f2445dab640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f24401369e0 0x7f24401e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48916/0 (socket says 172.21.15.46:48916) 2024-03-25T18:09:26.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.788+0000 7f2445dab640 1 -- 172.21.15.46:0/3868165850 learned_addr learned my addr 172.21.15.46:0/3868165850 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:26.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.791+0000 7f2435ffb640 10 client.?.objecter ms_handle_connect 0x7f24400a98f0 2024-03-25T18:09:26.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.791+0000 7f2435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:26.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.794+0000 7f2435ffb640 10 client.?.objecter ms_handle_connect 0x7f24401369e0 2024-03-25T18:09:26.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.794+0000 7f2435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:26.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.802+0000 7f2435ffb640 10 client.?.objecter ms_handle_connect 0x7f24401430c0 2024-03-25T18:09:26.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.802+0000 7f2435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899566009 0 0) 0x7f244013e9e0 con 0x7f24400a98f0 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:09:26.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.803+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424004eb0 con 0x7f24400a98f0 2024-03-25T18:09:26.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273838296 0 0) 0x7f2440137480 con 0x7f24401369e0 2024-03-25T18:09:26.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:09:26.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.805+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244013e9e0 con 0x7f24401369e0 2024-03-25T18:09:26.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2646307667 0 0) 0x7f2424004eb0 con 0x7f24400a98f0 2024-03-25T18:09:26.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2440137480 con 0x7f24400a98f0 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f243c001970 con 0x7f24400a98f0 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:26.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.815+0000 7f2435ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246457661 0 0) 0x7f244013eb70 con 0x7f24401430c0 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:09:26.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.817+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424004eb0 con 0x7f24401430c0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1302132522 0 0) 0x7f2440137480 con 0x7f24400a98f0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 >> v1:172.21.15.46:6790/0 conn(0x7f24401430c0 legacy=0x7f2440137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 >> v1:172.21.15.46:6789/0 conn(0x7f24401369e0 legacy=0x7f24401e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 1 monclient: found mon.b 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244013ed00 con 0x7f24400a98f0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:26.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f2435ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 5 monclient: authenticate success, global_id 4255 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 10 monclient: _renew_subs 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2440139440 con 0x7f24400a98f0 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 10 monclient: _renew_subs 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24401e3760 con 0x7f24400a98f0 2024-03-25T18:09:26.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.823+0000 7f244780e640 1 librados: init done 2024-03-25T18:09:26.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.825+0000 7f244780e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:26.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.825+0000 7f244780e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:26.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.825+0000 7f244780e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:26.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:26.825+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24400b3ef0 con 0x7f24400a98f0 2024-03-25T18:09:27.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f243c0047e0 con 0x7f24400a98f0 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f243c004af0 con 0x7f24400a98f0 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f2435ffb640 10 monclient: dump: 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:27.104 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:27.105 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:27.105 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:27.105 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:27.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.103+0000 7f24377fe640 4 set_mon_vals no callback set 2024-03-25T18:09:27.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.104+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f243c01aac0 con 0x7f24400a98f0 2024-03-25T18:09:27.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.104+0000 7f2444da9640 10 monclient: get_auth_request con 0x7f242406de00 auth_method 0 2024-03-25T18:09:27.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.105+0000 7f2435ffb640 10 client.4255.objecter ms_handle_connect 0x7f242406de00 2024-03-25T18:09:27.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.110+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f243c054860 con 0x7f24400a98f0 2024-03-25T18:09:27.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.110+0000 7f2435ffb640 10 client.4255.objecter ms_dispatch 0x7f2440000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:27.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.110+0000 7f2435ffb640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:27.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.110+0000 7f2435ffb640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:27.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.110+0000 7f2435ffb640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T18:09:27.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.112+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f243c0501a0 con 0x7f24400a98f0 2024-03-25T18:09:27.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.112+0000 7f2435ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:27.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.112+0000 7f2435ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:27.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.366+0000 7f244780e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:27.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.366+0000 7f244780e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:27.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.366+0000 7f244780e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:27.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.366+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24400627b0 con 0x7f24400a98f0 2024-03-25T18:09:27.391 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.390+0000 7f2435ffb640 1 -- 172.21.15.46:0/3868165850 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f243c0522a0 con 0x7f24400a98f0 2024-03-25T18:09:27.391 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.390+0000 7f2435ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:27.391 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.390+0000 7f2435ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:27.391 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:27.391 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 10 librados: watch_flush enter 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 10 librados: watch_flush exit 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 10 client.4255.objecter successfully canceled tick 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 >> v1:172.21.15.164:6800/1612084612 conn(0x7f242406de00 legacy=0x7f24240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 10 monclient: shutdown 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 >> v1:172.21.15.164:6789/0 conn(0x7f24400a98f0 legacy=0x7f24401dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 shutdown_connections 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 >> 172.21.15.46:0/3868165850 conn(0x7f24400b7ea0 msgr2=0x7f2440074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 shutdown_connections 2024-03-25T18:09:27.394 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 -- 172.21.15.46:0/3868165850 wait complete. 2024-03-25T18:09:27.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:27.393+0000 7f244780e640 1 librados: shutdown 2024-03-25T18:09:27.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:34.212 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:34.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.329+0000 7f0a38ce2640 10 monclient: get_monmap_and_config 2024-03-25T18:09:34.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.329+0000 7f0a38ce2640 10 monclient: build_initial_monmap 2024-03-25T18:09:34.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.329+0000 7f0a38ce2640 10 monclient: monmap: 2024-03-25T18:09:34.329 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:34.329 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:34.329790+0000 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:34.329790+0000 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:34.330 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 1 Processor -- start 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 1 -- start start 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient: init 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f0a34061830 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:34.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f0a341400e0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.330+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f0a341499d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.331+0000 7f0a38ce2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a34072600 con 0x7f0a34061830 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.331+0000 7f0a38ce2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a34140c80 con 0x7f0a341499d0 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.331+0000 7f0a38ce2640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a34072830 con 0x7f0a341400e0 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.331+0000 7f0a38ce2640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:34.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.331+0000 7f0a38ce2640 10 monclient(hunting): authenticate will time out at 7283.592645s 2024-03-25T18:09:34.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.335+0000 7f0a32d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0a341499d0 0x7f0a3414bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:54130/0 (socket says 172.21.15.46:54130) 2024-03-25T18:09:34.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.335+0000 7f0a32d9d640 1 -- 172.21.15.46:0/2639531762 learned_addr learned my addr 172.21.15.46:0/2639531762 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:34.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776642014 0 0) 0x7f0a34140c80 con 0x7f0a341499d0 2024-03-25T18:09:34.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:09:34.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.345+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20003520 con 0x7f0a341499d0 2024-03-25T18:09:34.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3363574843 0 0) 0x7f0a20003520 con 0x7f0a341499d0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 >> v1:172.21.15.46:6790/0 conn(0x7f0a341400e0 legacy=0x7f0a34140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 >> v1:172.21.15.46:6789/0 conn(0x7f0a34061830 legacy=0x7f0a3413d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 monclient: found mon.noname-c 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a34072b10 con 0x7f0a341499d0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a2c001a10 con 0x7f0a341499d0 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:34.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a31d9b640 10 monclient: dump: 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a38ce2640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:09:34.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.353+0000 7f0a38ce2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.365+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a2c001b70 con 0x7f0a341499d0 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a31d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a31d9b640 1 -- 172.21.15.46:0/2639531762 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a2c001f80 con 0x7f0a341499d0 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a31d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:34.366 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a31d9b640 10 monclient: dump: 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a38ce2640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:34.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a38ce2640 4 set_mon_vals no callback set 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a38ce2640 10 monclient: shutdown 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a38ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.366+0000 7f0a38ce2640 1 -- 172.21.15.46:0/2639531762 >> v1:172.21.15.164:6789/0 conn(0x7f0a341499d0 legacy=0x7f0a3414bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.367+0000 7f0a38ce2640 1 -- 172.21.15.46:0/2639531762 shutdown_connections 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.367+0000 7f0a38ce2640 1 -- 172.21.15.46:0/2639531762 >> 172.21.15.46:0/2639531762 conn(0x7f0a340b0ef0 msgr2=0x7f0a340b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.367+0000 7f0a38ce2640 1 -- 172.21.15.46:0/2639531762 shutdown_connections 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.367+0000 7f0a38ce2640 1 -- 172.21.15.46:0/2639531762 wait complete. 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 10 monclient: build_initial_monmap 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 10 monclient: monmap: 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:34.369 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 librados: starting msgr at 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 librados: starting objecter 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 Processor -- start 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 -- start start 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 librados: setting wanted keys 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 1 librados: calling monclient init 2024-03-25T18:09:34.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.368+0000 7f0a38ce2640 10 monclient: init 2024-03-25T18:09:34.371 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f0a34061830 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f0a341400e0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f0a341499d0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a34147430 con 0x7f0a34061830 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a341dac70 con 0x7f0a341400e0 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a341db8a0 con 0x7f0a341499d0 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:34.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.369+0000 7f0a38ce2640 10 monclient(hunting): authenticate will time out at 7283.631074s 2024-03-25T18:09:34.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.373+0000 7f0a32d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f0a341499d0 0x7f0a3413bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:47594/0 (socket says 172.21.15.46:47594) 2024-03-25T18:09:34.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.373+0000 7f0a32d9d640 1 -- 172.21.15.46:0/4062092574 learned_addr learned my addr 172.21.15.46:0/4062092574 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:34.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.377+0000 7f0a2affd640 10 client.?.objecter ms_handle_connect 0x7f0a34061830 2024-03-25T18:09:34.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.377+0000 7f0a2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.378 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.378+0000 7f0a2affd640 10 client.?.objecter ms_handle_connect 0x7f0a341499d0 2024-03-25T18:09:34.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.378+0000 7f0a2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.379+0000 7f0a2affd640 10 client.?.objecter ms_handle_connect 0x7f0a341400e0 2024-03-25T18:09:34.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.379+0000 7f0a2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151384975 0 0) 0x7f0a341db8a0 con 0x7f0a341499d0 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:09:34.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.384+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a14004eb0 con 0x7f0a341499d0 2024-03-25T18:09:34.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142726268 0 0) 0x7f0a14004eb0 con 0x7f0a341499d0 2024-03-25T18:09:34.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a341db8a0 con 0x7f0a341499d0 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a2c001f40 con 0x7f0a341499d0 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 10 monclient(hunting): dump: 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.406+0000 7f0a2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:34.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4276793680 0 0) 0x7f0a341db8a0 con 0x7f0a341499d0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 >> v1:172.21.15.164:6789/0 conn(0x7f0a341400e0 legacy=0x7f0a34145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 >> v1:172.21.15.46:6789/0 conn(0x7f0a34061830 legacy=0x7f0a341db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 1 monclient: found mon.c 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a341462d0 con 0x7f0a341499d0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 10 monclient: _finish_auth 0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a38ce2640 5 monclient: authenticate success, global_id 4262 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a38ce2640 10 monclient: _renew_subs 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a38ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.463+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a341d74e0 con 0x7f0a341499d0 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.464+0000 7f0a38ce2640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:34.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.464+0000 7f0a38ce2640 10 monclient: _renew_subs 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.464+0000 7f0a38ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.464+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3413d810 con 0x7f0a341499d0 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.464+0000 7f0a38ce2640 1 librados: init done 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.465+0000 7f0a38ce2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.465+0000 7f0a38ce2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.465+0000 7f0a38ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:34.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.465+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a341dae00 con 0x7f0a341499d0 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a2c004e10 con 0x7f0a341499d0 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a2c005120 con 0x7f0a341499d0 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a2affd640 10 monclient: dump: 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:34.726 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.725+0000 7f0a3159a640 4 set_mon_vals no callback set 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.726+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f0a2c01b0f0 con 0x7f0a341499d0 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.726+0000 7f0a3259c640 10 monclient: get_auth_request con 0x7f0a1406dc90 auth_method 0 2024-03-25T18:09:34.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.727+0000 7f0a2affd640 10 client.4262.objecter ms_handle_connect 0x7f0a1406dc90 2024-03-25T18:09:34.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f0a2c054c50 con 0x7f0a341499d0 2024-03-25T18:09:34.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 10 client.4262.objecter ms_dispatch 0x7f0a34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:34.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:34.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:34.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T18:09:34.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a2c050760 con 0x7f0a341499d0 2024-03-25T18:09:34.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:34.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.731+0000 7f0a2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:34.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.976+0000 7f0a38ce2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:34.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.976+0000 7f0a38ce2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:34.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.976+0000 7f0a38ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:34.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:34.976+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a340627b0 con 0x7f0a341499d0 2024-03-25T18:09:35.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.039+0000 7f0a2affd640 1 -- 172.21.15.46:0/4062092574 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f0a2c052690 con 0x7f0a341499d0 2024-03-25T18:09:35.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.039+0000 7f0a2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:35.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.039+0000 7f0a2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:35.039 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:35.040 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:35.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 10 librados: watch_flush enter 2024-03-25T18:09:35.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 10 librados: watch_flush exit 2024-03-25T18:09:35.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T18:09:35.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 10 client.4262.objecter successfully canceled tick 2024-03-25T18:09:35.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 >> v1:172.21.15.164:6800/1612084612 conn(0x7f0a1406dc90 legacy=0x7f0a14070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 10 monclient: shutdown 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 >> v1:172.21.15.46:6790/0 conn(0x7f0a341499d0 legacy=0x7f0a3413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 shutdown_connections 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 >> 172.21.15.46:0/4062092574 conn(0x7f0a340b0ef0 msgr2=0x7f0a340b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.041+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 shutdown_connections 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.042+0000 7f0a38ce2640 1 -- 172.21.15.46:0/4062092574 wait complete. 2024-03-25T18:09:35.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:35.042+0000 7f0a38ce2640 1 librados: shutdown 2024-03-25T18:09:35.054 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:38.508 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:14.426 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.605 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:14.523 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.705 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:14.623 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.765 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:14.682 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.846 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:14.764 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:38.995 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:14.866 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:39.006 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:14.924 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:39.054 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:14.925 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:39.291 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:15.162 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:41.860 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.973+0000 7fb9173fb640 10 monclient: get_monmap_and_config 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.973+0000 7fb9173fb640 10 monclient: build_initial_monmap 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.973+0000 7fb9173fb640 10 monclient: monmap: 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:41.974078+0000 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:41.974078+0000 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:41.974 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:41.975 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:41.975 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:41.975 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.974+0000 7fb9173fb640 1 Processor -- start 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.974+0000 7fb9173fb640 1 -- start start 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.974+0000 7fb9173fb640 10 monclient: init 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-b con 0x7fb9100a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-c con 0x7fb91013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:41.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-a con 0x7fb9101430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:41.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb910142520 con 0x7fb9101430c0 2024-03-25T18:09:41.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100b28a0 con 0x7fb91013f710 2024-03-25T18:09:41.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100b2a30 con 0x7fb9100a98f0 2024-03-25T18:09:41.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:41.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.975+0000 7fb9173fb640 10 monclient(hunting): authenticate will time out at 7291.236964s 2024-03-25T18:09:41.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.983+0000 7fb915197640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fb9100a98f0 0x7fb9100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:51684/0 (socket says 172.21.15.46:51684) 2024-03-25T18:09:41.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.983+0000 7fb915197640 1 -- 172.21.15.46:0/1071359901 learned_addr learned my addr 172.21.15.46:0/1071359901 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609978327 0 0) 0x7fb9100b2a30 con 0x7fb9100a98f0 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:09:42.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:41.999+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003520 con 0x7fb9100a98f0 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571977251 0 0) 0x7fb9100b28a0 con 0x7fb91013f710 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:09:42.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.001+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9100b2a30 con 0x7fb91013f710 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949579851 0 0) 0x7fb8fc003520 con 0x7fb9100a98f0 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 >> v1:172.21.15.164:6789/0 conn(0x7fb91013f710 legacy=0x7fb9101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 >> v1:172.21.15.46:6789/0 conn(0x7fb9101430c0 legacy=0x7fb91014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 monclient: found mon.noname-b 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100b44c0 con 0x7fb9100a98f0 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:42.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb90c001690 con 0x7fb9100a98f0 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb907fff640 10 monclient: dump: 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:42.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb9173fb640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:09:42.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.006+0000 7fb9173fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb90c001870 con 0x7fb9100a98f0 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 1 -- 172.21.15.46:0/1071359901 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb90c001c80 con 0x7fb9100a98f0 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb907fff640 10 monclient: dump: 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb9173fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.019+0000 7fb9173fb640 4 set_mon_vals no callback set 2024-03-25T18:09:42.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.020+0000 7fb9173fb640 10 monclient: shutdown 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.020+0000 7fb9173fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.020+0000 7fb9173fb640 1 -- 172.21.15.46:0/1071359901 >> v1:172.21.15.46:6790/0 conn(0x7fb9100a98f0 legacy=0x7fb9100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.020+0000 7fb9173fb640 1 -- 172.21.15.46:0/1071359901 shutdown_connections 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.020+0000 7fb9173fb640 1 -- 172.21.15.46:0/1071359901 >> 172.21.15.46:0/1071359901 conn(0x7fb9100b7ea0 msgr2=0x7fb9100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.021+0000 7fb9173fb640 1 -- 172.21.15.46:0/1071359901 shutdown_connections 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.021+0000 7fb9173fb640 1 -- 172.21.15.46:0/1071359901 wait complete. 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.021+0000 7fb9173fb640 10 monclient: build_initial_monmap 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 10 monclient: monmap: 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:42.023 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 librados: starting msgr at 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 librados: starting objecter 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 Processor -- start 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 -- start start 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 librados: setting wanted keys 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 1 librados: calling monclient init 2024-03-25T18:09:42.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.022+0000 7fb9173fb640 10 monclient: init 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-b con 0x7fb9100a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-c con 0x7fb91013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient(hunting): picked mon.noname-a con 0x7fb9101430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9101374f0 con 0x7fb9101430c0 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9101db360 con 0x7fb9100a98f0 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb910133a40 con 0x7fb91013f710 2024-03-25T18:09:42.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:42.026 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.023+0000 7fb9173fb640 10 monclient(hunting): authenticate will time out at 7291.285074s 2024-03-25T18:09:42.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.026+0000 7fb915197640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb9100a98f0 0x7fb9101dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:58052/0 (socket says 172.21.15.46:58052) 2024-03-25T18:09:42.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.026+0000 7fb915197640 1 -- 172.21.15.46:0/2966740082 learned_addr learned my addr 172.21.15.46:0/2966740082 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:42.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.027+0000 7fb905ffb640 10 client.?.objecter ms_handle_connect 0x7fb9100a98f0 2024-03-25T18:09:42.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.027+0000 7fb905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:42.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.030+0000 7fb905ffb640 10 client.?.objecter ms_handle_connect 0x7fb9101430c0 2024-03-25T18:09:42.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.030+0000 7fb905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:42.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.036+0000 7fb905ffb640 10 client.?.objecter ms_handle_connect 0x7fb91013f710 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.036+0000 7fb905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241094720 0 0) 0x7fb9101db360 con 0x7fb9100a98f0 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f4004eb0 con 0x7fb9100a98f0 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128177369 0 0) 0x7fb9101374f0 con 0x7fb9101430c0 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:09:42.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.037+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9101db360 con 0x7fb9101430c0 2024-03-25T18:09:42.048 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1470017940 0 0) 0x7fb8f4004eb0 con 0x7fb9100a98f0 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9101374f0 con 0x7fb9100a98f0 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb90c001c40 con 0x7fb9100a98f0 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:42.049 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:42.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.048+0000 7fb905ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2802947593 0 0) 0x7fb9101374f0 con 0x7fb9100a98f0 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 >> v1:172.21.15.46:6790/0 conn(0x7fb91013f710 legacy=0x7fb910132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 >> v1:172.21.15.46:6789/0 conn(0x7fb9101430c0 legacy=0x7fb910133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 1 monclient: found mon.b 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910138000 con 0x7fb9100a98f0 2024-03-25T18:09:42.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 10 monclient: _renew_subs 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb910136030 con 0x7fb9100a98f0 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 10 monclient: _renew_subs 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9100b7570 con 0x7fb9100a98f0 2024-03-25T18:09:42.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.057+0000 7fb9173fb640 1 librados: init done 2024-03-25T18:09:42.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.058+0000 7fb9173fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:42.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.058+0000 7fb9173fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:42.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.058+0000 7fb9173fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:42.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.058+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9100627b0 con 0x7fb9100a98f0 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb90c001170 con 0x7fb9100a98f0 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb90c004d90 con 0x7fb9100a98f0 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb905ffb640 10 monclient: dump: 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:42.327 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.326+0000 7fb9077fe640 4 set_mon_vals no callback set 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.327+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb90c01ade0 con 0x7fb9100a98f0 2024-03-25T18:09:42.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.328+0000 7fb914996640 10 monclient: get_auth_request con 0x7fb8f406dcf0 auth_method 0 2024-03-25T18:09:42.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.328+0000 7fb905ffb640 10 client.4264.objecter ms_handle_connect 0x7fb8f406dcf0 2024-03-25T18:09:42.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb90c054b00 con 0x7fb9100a98f0 2024-03-25T18:09:42.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 10 client.4264.objecter ms_dispatch 0x7fb910000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb90c050440 con 0x7fb9100a98f0 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:42.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.333+0000 7fb905ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:42.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.562+0000 7fb9173fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.563+0000 7fb9173fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.563+0000 7fb9173fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:42.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.563+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb91013e210 con 0x7fb9100a98f0 2024-03-25T18:09:42.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.595+0000 7fb905ffb640 1 -- 172.21.15.46:0/2966740082 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb90c052540 con 0x7fb9100a98f0 2024-03-25T18:09:42.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.595+0000 7fb905ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.595+0000 7fb905ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:42.596 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:42.596 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 10 librados: watch_flush enter 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 10 librados: watch_flush exit 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb8f406dcf0 legacy=0x7fb8f40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 10 monclient: shutdown 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 >> v1:172.21.15.164:6789/0 conn(0x7fb9100a98f0 legacy=0x7fb9101dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 shutdown_connections 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 >> 172.21.15.46:0/2966740082 conn(0x7fb9100b7ea0 msgr2=0x7fb91013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 shutdown_connections 2024-03-25T18:09:42.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 -- 172.21.15.46:0/2966740082 wait complete. 2024-03-25T18:09:42.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:42.598+0000 7fb9173fb640 1 librados: shutdown 2024-03-25T18:09:42.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:49.415 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:49.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.531+0000 7efdcfe74640 10 monclient: get_monmap_and_config 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.531+0000 7efdcfe74640 10 monclient: build_initial_monmap 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.531+0000 7efdcfe74640 10 monclient: monmap: 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:49.532153+0000 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:49.532153+0000 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:49.532 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.532+0000 7efdcfe74640 1 Processor -- start 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.532+0000 7efdcfe74640 1 -- start start 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.532+0000 7efdcfe74640 10 monclient: init 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-a con 0x7efdc80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-b con 0x7efdc813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-c con 0x7efdc81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8142520 con 0x7efdc80a98f0 2024-03-25T18:09:49.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80b28a0 con 0x7efdc81430c0 2024-03-25T18:09:49.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80b2a30 con 0x7efdc813f710 2024-03-25T18:09:49.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:49.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.533+0000 7efdcfe74640 10 monclient(hunting): authenticate will time out at 7298.794991s 2024-03-25T18:09:49.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.546+0000 7efdcd40f640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7efdc813f710 0x7efdc81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:54226/0 (socket says 172.21.15.46:54226) 2024-03-25T18:09:49.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.546+0000 7efdcd40f640 1 -- 172.21.15.46:0/3125330940 learned_addr learned my addr 172.21.15.46:0/3125330940 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413762921 0 0) 0x7efdc80b2a30 con 0x7efdc813f710 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:09:49.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.567+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb4003520 con 0x7efdc813f710 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492567353 0 0) 0x7efdc80b28a0 con 0x7efdc81430c0 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:09:49.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.571+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc80b2a30 con 0x7efdc81430c0 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11593415 0 0) 0x7efdc8142520 con 0x7efdc80a98f0 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:09:49.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.573+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc80b28a0 con 0x7efdc80a98f0 2024-03-25T18:09:49.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.579+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2983713823 0 0) 0x7efdb4003520 con 0x7efdc813f710 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 >> v1:172.21.15.164:6789/0 conn(0x7efdc81430c0 legacy=0x7efdc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 >> v1:172.21.15.46:6789/0 conn(0x7efdc80a98f0 legacy=0x7efdc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: _finish_hunting 0 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 1 monclient: found mon.noname-b 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80b44c0 con 0x7efdc813f710 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: _finish_auth 0 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: _check_auth_tickets 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efdb8001690 con 0x7efdc813f710 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdcfe74640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdcfe74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:49.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdccc0e640 10 monclient: dump: 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:49.582 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:49.583 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.580+0000 7efdcfe74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:49.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb80017f0 con 0x7efdc813f710 2024-03-25T18:09:49.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 1 -- 172.21.15.46:0/3125330940 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efdb8001c00 con 0x7efdc813f710 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdccc0e640 10 monclient: dump: 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdcfe74640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:49.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.595+0000 7efdcfe74640 4 set_mon_vals no callback set 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 10 monclient: shutdown 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 1 -- 172.21.15.46:0/3125330940 >> v1:172.21.15.46:6790/0 conn(0x7efdc813f710 legacy=0x7efdc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 1 -- 172.21.15.46:0/3125330940 shutdown_connections 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 1 -- 172.21.15.46:0/3125330940 >> 172.21.15.46:0/3125330940 conn(0x7efdc80b7ea0 msgr2=0x7efdc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:49.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 1 -- 172.21.15.46:0/3125330940 shutdown_connections 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.596+0000 7efdcfe74640 1 -- 172.21.15.46:0/3125330940 wait complete. 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.597+0000 7efdcfe74640 10 monclient: build_initial_monmap 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.597+0000 7efdcfe74640 10 monclient: monmap: 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:49.599 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.597+0000 7efdcfe74640 1 librados: starting msgr at 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.597+0000 7efdcfe74640 1 librados: starting objecter 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.597+0000 7efdcfe74640 1 Processor -- start 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 -- start start 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 librados: setting wanted keys 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 librados: calling monclient init 2024-03-25T18:09:49.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient: init 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-b con 0x7efdc80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-c con 0x7efdc813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 10 monclient(hunting): picked mon.noname-a con 0x7efdc81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80ab8b0 con 0x7efdc81430c0 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80abb90 con 0x7efdc80a98f0 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.598+0000 7efdcfe74640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8137870 con 0x7efdc813f710 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.599+0000 7efdcfe74640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:49.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.599+0000 7efdcfe74640 10 monclient(hunting): authenticate will time out at 7298.860548s 2024-03-25T18:09:49.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.605+0000 7efdcdc10640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efdc80a98f0 0x7efdc81350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48292/0 (socket says 172.21.15.46:48292) 2024-03-25T18:09:49.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.605+0000 7efdcdc10640 1 -- 172.21.15.46:0/3378308567 learned_addr learned my addr 172.21.15.46:0/3378308567 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:49.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.606+0000 7efdbe7fc640 10 client.?.objecter ms_handle_connect 0x7efdc80a98f0 2024-03-25T18:09:49.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.606+0000 7efdbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.607+0000 7efdbe7fc640 10 client.?.objecter ms_handle_connect 0x7efdc81430c0 2024-03-25T18:09:49.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.607+0000 7efdbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.616+0000 7efdbe7fc640 10 client.?.objecter ms_handle_connect 0x7efdc813f710 2024-03-25T18:09:49.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.616+0000 7efdbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222064595 0 0) 0x7efdc80abb90 con 0x7efdc80a98f0 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:09:49.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.625+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac004eb0 con 0x7efdc80a98f0 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547091860 0 0) 0x7efdc80ab8b0 con 0x7efdc81430c0 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:09:49.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.627+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc80abb90 con 0x7efdc81430c0 2024-03-25T18:09:49.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554965286 0 0) 0x7efdc8137870 con 0x7efdc813f710 2024-03-25T18:09:49.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:09:49.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.634+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc80ab8b0 con 0x7efdc813f710 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720620932 0 0) 0x7efdc80abb90 con 0x7efdc81430c0 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc8137870 con 0x7efdc81430c0 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efdc0001b50 con 0x7efdc81430c0 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:49.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.645+0000 7efdbe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3466382027 0 0) 0x7efdac004eb0 con 0x7efdc80a98f0 2024-03-25T18:09:49.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc80abb90 con 0x7efdc80a98f0 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efdc4001a10 con 0x7efdc80a98f0 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.646+0000 7efdbe7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:49.648 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2683804389 0 0) 0x7efdc80abb90 con 0x7efdc80a98f0 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 >> v1:172.21.15.46:6790/0 conn(0x7efdc813f710 legacy=0x7efdc81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 >> v1:172.21.15.46:6789/0 conn(0x7efdc81430c0 legacy=0x7efdc81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 1 monclient: found mon.b 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc81389e0 con 0x7efdc80a98f0 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:49.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 10 monclient: _renew_subs 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc8136d90 con 0x7efdc80a98f0 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 10 monclient: _renew_subs 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc81384c0 con 0x7efdc80a98f0 2024-03-25T18:09:49.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.658+0000 7efdcfe74640 1 librados: init done 2024-03-25T18:09:49.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.659+0000 7efdcfe74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:49.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.659+0000 7efdcfe74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:49.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.659+0000 7efdcfe74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:49.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.659+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc80627b0 con 0x7efdc80a98f0 2024-03-25T18:09:49.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc4001c70 con 0x7efdc80a98f0 2024-03-25T18:09:49.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7efdc4003a60 con 0x7efdc80a98f0 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbe7fc640 10 monclient: dump: 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:49.957 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:49.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.955+0000 7efdbffff640 4 set_mon_vals no callback set 2024-03-25T18:09:49.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.956+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7efdc4003c80 con 0x7efdc80a98f0 2024-03-25T18:09:49.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.957+0000 7efdcd40f640 10 monclient: get_auth_request con 0x7efdac06dfb0 auth_method 0 2024-03-25T18:09:49.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.957+0000 7efdbe7fc640 10 client.4270.objecter ms_handle_connect 0x7efdac06dfb0 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.960+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7efdc40548b0 con 0x7efdc80a98f0 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.960+0000 7efdbe7fc640 10 client.4270.objecter ms_dispatch 0x7efdc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.960+0000 7efdbe7fc640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.960+0000 7efdbe7fc640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.960+0000 7efdbe7fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.961+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdc40503c0 con 0x7efdc80a98f0 2024-03-25T18:09:49.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.961+0000 7efdbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:49.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:49.961+0000 7efdbe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:50.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.200+0000 7efdcfe74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.200+0000 7efdcfe74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.200+0000 7efdcfe74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:09:50.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.200+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc81434a0 con 0x7efdc80a98f0 2024-03-25T18:09:50.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.236+0000 7efdbe7fc640 1 -- 172.21.15.46:0/3378308567 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7efdc40522f0 con 0x7efdc80a98f0 2024-03-25T18:09:50.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.236+0000 7efdbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.236+0000 7efdbe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:50.237 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:50.237 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 10 librados: watch_flush enter 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 10 librados: watch_flush exit 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 >> v1:172.21.15.164:6800/1612084612 conn(0x7efdac06dfb0 legacy=0x7efdac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 10 monclient: shutdown 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 >> v1:172.21.15.164:6789/0 conn(0x7efdc80a98f0 legacy=0x7efdc81350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 shutdown_connections 2024-03-25T18:09:50.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 >> 172.21.15.46:0/3378308567 conn(0x7efdc80b7ea0 msgr2=0x7efdc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:50.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 shutdown_connections 2024-03-25T18:09:50.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 -- 172.21.15.46:0/3378308567 wait complete. 2024-03-25T18:09:50.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:50.239+0000 7efdcfe74640 1 librados: shutdown 2024-03-25T18:09:50.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:57.058 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.170+0000 7f154505c640 10 monclient: get_monmap_and_config 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.170+0000 7f154505c640 10 monclient: build_initial_monmap 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.170+0000 7f154505c640 10 monclient: monmap: 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:09:57.171259+0000 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:09:57.171259+0000 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:57.171 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:09:57.172 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:09:57.172 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 1 Processor -- start 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 1 -- start start 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 10 monclient: init 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 10 monclient(hunting): picked mon.noname-a con 0x7f15400a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.171+0000 7f154505c640 10 monclient(hunting): picked mon.noname-c con 0x7f154013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 10 monclient(hunting): picked mon.noname-b con 0x7f15401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1540142520 con 0x7f15400a98f0 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15400b28a0 con 0x7f154013f710 2024-03-25T18:09:57.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15400b2a30 con 0x7f15401430c0 2024-03-25T18:09:57.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:57.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.172+0000 7f154505c640 10 monclient(hunting): authenticate will time out at 7306.433617s 2024-03-25T18:09:57.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.181+0000 7f153f59e640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f15401430c0 0x7f154014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:41170/0 (socket says 172.21.15.46:41170) 2024-03-25T18:09:57.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.181+0000 7f153f59e640 1 -- 172.21.15.46:0/318074152 learned_addr learned my addr 172.21.15.46:0/318074152 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784028370 0 0) 0x7f15400b28a0 con 0x7f154013f710 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:09:57.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.196+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1528003520 con 0x7f154013f710 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740338088 0 0) 0x7f15400b2a30 con 0x7f15401430c0 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.197+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15400b28a0 con 0x7f15401430c0 2024-03-25T18:09:57.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528915720 0 0) 0x7f1528003520 con 0x7f154013f710 2024-03-25T18:09:57.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 >> v1:172.21.15.46:6790/0 conn(0x7f15401430c0 legacy=0x7f154014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 >> v1:172.21.15.46:6789/0 conn(0x7f15400a98f0 legacy=0x7f15400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15400b44c0 con 0x7f154013f710 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1534001690 con 0x7f154013f710 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f153dd9b640 10 monclient: dump: 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.218 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f154505c640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:09:57.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.216+0000 7f154505c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:57.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15340017f0 con 0x7f154013f710 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 1 -- 172.21.15.46:0/318074152 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1534001c00 con 0x7f154013f710 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f154505c640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f154505c640 4 set_mon_vals no callback set 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.231+0000 7f153dd9b640 10 monclient: dump: 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.232 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 10 monclient: shutdown 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 1 -- 172.21.15.46:0/318074152 >> v1:172.21.15.164:6789/0 conn(0x7f154013f710 legacy=0x7f15401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 1 -- 172.21.15.46:0/318074152 shutdown_connections 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 1 -- 172.21.15.46:0/318074152 >> 172.21.15.46:0/318074152 conn(0x7f15400b7ea0 msgr2=0x7f15400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 1 -- 172.21.15.46:0/318074152 shutdown_connections 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.232+0000 7f154505c640 1 -- 172.21.15.46:0/318074152 wait complete. 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.233+0000 7f154505c640 10 monclient: build_initial_monmap 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.233+0000 7f154505c640 10 monclient: monmap: 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 librados: starting msgr at 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 librados: starting objecter 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 Processor -- start 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 -- start start 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 librados: setting wanted keys 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 1 librados: calling monclient init 2024-03-25T18:09:57.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.234+0000 7f154505c640 10 monclient: init 2024-03-25T18:09:57.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:57.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:57.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient(hunting): picked mon.noname-c con 0x7f15400a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:09:57.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient(hunting): picked mon.noname-b con 0x7f154013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:09:57.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient(hunting): picked mon.noname-a con 0x7f15401430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1540139070 con 0x7f15401430c0 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1540133710 con 0x7f154013f710 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15401338a0 con 0x7f15400a98f0 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.235+0000 7f154505c640 10 monclient(hunting): authenticate will time out at 7306.497093s 2024-03-25T18:09:57.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.249+0000 7f153e59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f154013f710 0x7f15401daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34304/0 (socket says 172.21.15.46:34304) 2024-03-25T18:09:57.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.249+0000 7f153e59c640 1 -- 172.21.15.46:0/2847231239 learned_addr learned my addr 172.21.15.46:0/2847231239 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:09:57.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.253+0000 7f15277fe640 10 client.?.objecter ms_handle_connect 0x7f154013f710 2024-03-25T18:09:57.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.253+0000 7f15277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.259+0000 7f15277fe640 10 client.?.objecter ms_handle_connect 0x7f15400a98f0 2024-03-25T18:09:57.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.259+0000 7f15277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.265+0000 7f15277fe640 10 client.?.objecter ms_handle_connect 0x7f15401430c0 2024-03-25T18:09:57.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.265+0000 7f15277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096600516 0 0) 0x7f1540133710 con 0x7f154013f710 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:09:57.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.272+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c004eb0 con 0x7f154013f710 2024-03-25T18:09:57.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098367285 0 0) 0x7f15401338a0 con 0x7f15400a98f0 2024-03-25T18:09:57.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:09:57.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.275+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540133710 con 0x7f15400a98f0 2024-03-25T18:09:57.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476329051 0 0) 0x7f1540139070 con 0x7f15401430c0 2024-03-25T18:09:57.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:09:57.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.278+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15401338a0 con 0x7f15401430c0 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.283+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984180786 0 0) 0x7f152c004eb0 con 0x7f154013f710 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1540139070 con 0x7f154013f710 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1534001bc0 con 0x7f154013f710 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 10 monclient(hunting): dump: 2024-03-25T18:09:57.284 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.284+0000 7f15277fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:57.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980551223 0 0) 0x7f1540133710 con 0x7f15400a98f0 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f152c004eb0 con 0x7f15400a98f0 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f15300018b0 con 0x7f15400a98f0 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.286+0000 7f15277fe640 10 monclient(hunting): dump: 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.287 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.288 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397337064 0 0) 0x7f15401338a0 con 0x7f15401430c0 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1540133710 con 0x7f15401430c0 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1538001b90 con 0x7f15401430c0 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.293+0000 7f15277fe640 10 monclient(hunting): dump: 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.294 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.295 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.295 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.295 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.295 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.295 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1345950751 0 0) 0x7f152c004eb0 con 0x7f15400a98f0 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 >> v1:172.21.15.164:6789/0 conn(0x7f154013f710 legacy=0x7f15401daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 >> v1:172.21.15.46:6789/0 conn(0x7f15401430c0 legacy=0x7f1540133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 1 monclient: found mon.c 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540133a30 con 0x7f15400a98f0 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:57.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f15277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 5 monclient: authenticate success, global_id 4280 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 10 monclient: _renew_subs 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15400b7100 con 0x7f15400a98f0 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 10 monclient: _renew_subs 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1540136e60 con 0x7f15400a98f0 2024-03-25T18:09:57.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.301+0000 7f154505c640 1 librados: init done 2024-03-25T18:09:57.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.302+0000 7f154505c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.302+0000 7f154505c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.302+0000 7f154505c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:57.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.302+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1540074cc0 con 0x7f15400a98f0 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15300035f0 con 0x7f15400a98f0 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1530003900 con 0x7f15400a98f0 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f15277fe640 10 monclient: dump: 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.625 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.624+0000 7f153d59a640 4 set_mon_vals no callback set 2024-03-25T18:09:57.626 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.625+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f15300198d0 con 0x7f15400a98f0 2024-03-25T18:09:57.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.626+0000 7f153e59c640 10 monclient: get_auth_request con 0x7f152c06e160 auth_method 0 2024-03-25T18:09:57.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.626+0000 7f15277fe640 10 client.4280.objecter ms_handle_connect 0x7f152c06e160 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f1530053520 con 0x7f15400a98f0 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 10 client.4280.objecter ms_dispatch 0x7f1540000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f153004f030 con 0x7f15400a98f0 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.631+0000 7f15277fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:57.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.878+0000 7f154505c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.878+0000 7f154505c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.878+0000 7f154505c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:09:57.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.878+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15400627b0 con 0x7f15400a98f0 2024-03-25T18:09:57.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.920+0000 7f15277fe640 1 -- 172.21.15.46:0/2847231239 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f1530050f60 con 0x7f15400a98f0 2024-03-25T18:09:57.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.920+0000 7f15277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.920+0000 7f15277fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:57.922 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:09:57.922 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 10 librados: watch_flush enter 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 10 librados: watch_flush exit 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 10 client.4280.objecter successfully canceled tick 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 >> v1:172.21.15.164:6800/1612084612 conn(0x7f152c06e160 legacy=0x7f152c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 10 monclient: shutdown 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.923+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 >> v1:172.21.15.46:6790/0 conn(0x7f15400a98f0 legacy=0x7f15401378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.924+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 shutdown_connections 2024-03-25T18:09:57.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.924+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 >> 172.21.15.46:0/2847231239 conn(0x7f15400b7ea0 msgr2=0x7f1540074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:57.926 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.924+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 shutdown_connections 2024-03-25T18:09:57.926 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.924+0000 7f154505c640 1 -- 172.21.15.46:0/2847231239 wait complete. 2024-03-25T18:09:57.926 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:09:57.924+0000 7f154505c640 1 librados: shutdown 2024-03-25T18:09:57.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:58.524 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:34.442 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.580 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:34.498 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.647 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:34.565 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.648 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:34.566 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.653 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:34.524 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.657 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:34.529 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.660 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:34.578 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.701 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:34.572 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:58.720 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:34.638 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.741 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:04.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.858+0000 7f7b4793a640 10 monclient: get_monmap_and_config 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.858+0000 7f7b4793a640 10 monclient: build_initial_monmap 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.858+0000 7f7b4793a640 10 monclient: monmap: 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:04.858850+0000 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:04.858850+0000 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.858+0000 7f7b4793a640 1 Processor -- start 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 1 -- start start 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient: init 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-a con 0x7f7b400a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-b con 0x7f7b401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40142520 con 0x7f7b400a98f0 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400b28a0 con 0x7f7b4013f710 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400b2a30 con 0x7f7b401430c0 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.859+0000 7f7b4793a640 10 monclient(hunting): authenticate will time out at 7314.121504s 2024-03-25T18:10:04.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.878+0000 7f7b44ed5640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7b4013f710 0x7f7b401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:40424/0 (socket says 172.21.15.46:40424) 2024-03-25T18:10:04.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.878+0000 7f7b44ed5640 1 -- 172.21.15.46:0/324718725 learned_addr learned my addr 172.21.15.46:0/324718725 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:04.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643889149 0 0) 0x7f7b400b28a0 con 0x7f7b4013f710 2024-03-25T18:10:04.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:10:04.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.893+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c003520 con 0x7f7b4013f710 2024-03-25T18:10:04.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923939973 0 0) 0x7f7b40142520 con 0x7f7b400a98f0 2024-03-25T18:10:04.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:10:04.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.898+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b400b28a0 con 0x7f7b400a98f0 2024-03-25T18:10:04.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362328965 0 0) 0x7f7b400b2a30 con 0x7f7b401430c0 2024-03-25T18:10:04.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:10:04.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.911+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b40142520 con 0x7f7b401430c0 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919378856 0 0) 0x7f7b2c003520 con 0x7f7b4013f710 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 >> v1:172.21.15.46:6790/0 conn(0x7f7b401430c0 legacy=0x7f7b4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 >> v1:172.21.15.46:6789/0 conn(0x7f7b400a98f0 legacy=0x7f7b400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 monclient: found mon.noname-c 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400b44c0 con 0x7f7b4013f710 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b30001690 con 0x7f7b4013f710 2024-03-25T18:10:04.917 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b37fff640 10 monclient: dump: 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b4793a640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:10:04.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.916+0000 7f7b4793a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:04.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b300017f0 con 0x7f7b4013f710 2024-03-25T18:10:04.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:04.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 1 -- 172.21.15.46:0/324718725 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b30001c00 con 0x7f7b4013f710 2024-03-25T18:10:04.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b37fff640 10 monclient: dump: 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b4793a640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:04.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b4793a640 4 set_mon_vals no callback set 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b4793a640 10 monclient: shutdown 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b4793a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.937+0000 7f7b4793a640 1 -- 172.21.15.46:0/324718725 >> v1:172.21.15.164:6789/0 conn(0x7f7b4013f710 legacy=0x7f7b401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.938+0000 7f7b4793a640 1 -- 172.21.15.46:0/324718725 shutdown_connections 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.938+0000 7f7b4793a640 1 -- 172.21.15.46:0/324718725 >> 172.21.15.46:0/324718725 conn(0x7f7b400b7ea0 msgr2=0x7f7b400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.938+0000 7f7b4793a640 1 -- 172.21.15.46:0/324718725 shutdown_connections 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.938+0000 7f7b4793a640 1 -- 172.21.15.46:0/324718725 wait complete. 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 10 monclient: build_initial_monmap 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 10 monclient: monmap: 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:04.941 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 librados: starting msgr at 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 librados: starting objecter 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 Processor -- start 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 -- start start 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 librados: setting wanted keys 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 1 librados: calling monclient init 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.939+0000 7f7b4793a640 10 monclient: init 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:04.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-a con 0x7f7b400a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient(hunting): picked mon.noname-c con 0x7f7b401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40138c10 con 0x7f7b400a98f0 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40138e60 con 0x7f7b4013f710 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40139140 con 0x7f7b401430c0 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:04.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.940+0000 7f7b4793a640 10 monclient(hunting): authenticate will time out at 7314.201871s 2024-03-25T18:10:04.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.954+0000 7f7b45ed7640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f7b401430c0 0x7f7b401e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:42354/0 (socket says 172.21.15.46:42354) 2024-03-25T18:10:04.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.954+0000 7f7b45ed7640 1 -- 172.21.15.46:0/2203183991 learned_addr learned my addr 172.21.15.46:0/2203183991 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:04.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.958+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b4013f710 2024-03-25T18:10:04.959 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.958+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.959+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b401430c0 2024-03-25T18:10:04.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.959+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.959+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b400a98f0 2024-03-25T18:10:04.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.959+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839034649 0 0) 0x7f7b40138e60 con 0x7f7b4013f710 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:10:04.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.970+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b24004eb0 con 0x7f7b4013f710 2024-03-25T18:10:04.972 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024366655 0 0) 0x7f7b40139140 con 0x7f7b401430c0 2024-03-25T18:10:04.972 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.972 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.972 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:10:04.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.972+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b40138e60 con 0x7f7b401430c0 2024-03-25T18:10:04.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169379657 0 0) 0x7f7b24004eb0 con 0x7f7b4013f710 2024-03-25T18:10:04.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b40139140 con 0x7f7b4013f710 2024-03-25T18:10:04.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b30001b00 con 0x7f7b4013f710 2024-03-25T18:10:04.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:04.984 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.982+0000 7f7b35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168927859 0 0) 0x7f7b40138c10 con 0x7f7b400a98f0 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:10:04.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.984+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b24004eb0 con 0x7f7b400a98f0 2024-03-25T18:10:04.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 798305575 0 0) 0x7f7b40139140 con 0x7f7b4013f710 2024-03-25T18:10:04.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 >> v1:172.21.15.46:6790/0 conn(0x7f7b401430c0 legacy=0x7f7b401e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 >> v1:172.21.15.46:6789/0 conn(0x7f7b400a98f0 legacy=0x7f7b40137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:04.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 1 monclient: found mon.b 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40139920 con 0x7f7b4013f710 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b4793a640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b4793a640 10 monclient: _renew_subs 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b4793a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.994+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b401382e0 con 0x7f7b4013f710 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.995+0000 7f7b4793a640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.995+0000 7f7b4793a640 10 monclient: _renew_subs 2024-03-25T18:10:04.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.995+0000 7f7b4793a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.995+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b400ab8b0 con 0x7f7b4013f710 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.995+0000 7f7b4793a640 1 librados: init done 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.996+0000 7f7b4793a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.996+0000 7f7b4793a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.996+0000 7f7b4793a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:04.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:04.996+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b40137a20 con 0x7f7b4013f710 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b30004a90 con 0x7f7b4013f710 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b30004da0 con 0x7f7b4013f710 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b35ffb640 10 monclient: dump: 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:05.265 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.264+0000 7f7b377fe640 4 set_mon_vals no callback set 2024-03-25T18:10:05.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.265+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7b3001ad70 con 0x7f7b4013f710 2024-03-25T18:10:05.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.266+0000 7f7b456d6640 10 monclient: get_auth_request con 0x7f7b2406ddb0 auth_method 0 2024-03-25T18:10:05.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.266+0000 7f7b35ffb640 10 client.4282.objecter ms_handle_connect 0x7f7b2406ddb0 2024-03-25T18:10:05.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7b30054860 con 0x7f7b4013f710 2024-03-25T18:10:05.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 10 client.4282.objecter ms_dispatch 0x7f7b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:05.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:05.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:05.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T18:10:05.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b30050370 con 0x7f7b4013f710 2024-03-25T18:10:05.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:05.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.274+0000 7f7b35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.506+0000 7f7b4793a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.506+0000 7f7b4793a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.506+0000 7f7b4793a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.506+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b4013d010 con 0x7f7b4013f710 2024-03-25T18:10:05.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.583+0000 7f7b35ffb640 1 -- 172.21.15.46:0/2203183991 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7b300522a0 con 0x7f7b4013f710 2024-03-25T18:10:05.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.584+0000 7f7b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.584+0000 7f7b35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:05.584 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:05.584 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 10 librados: watch_flush enter 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 10 librados: watch_flush exit 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 10 client.4282.objecter successfully canceled tick 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 >> v1:172.21.15.164:6800/1612084612 conn(0x7f7b2406ddb0 legacy=0x7f7b24070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 10 monclient: shutdown 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 >> v1:172.21.15.164:6789/0 conn(0x7f7b4013f710 legacy=0x7f7b40132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:05.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 shutdown_connections 2024-03-25T18:10:05.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 >> 172.21.15.46:0/2203183991 conn(0x7f7b400b7ea0 msgr2=0x7f7b40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:05.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 shutdown_connections 2024-03-25T18:10:05.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 -- 172.21.15.46:0/2203183991 wait complete. 2024-03-25T18:10:05.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:05.586+0000 7f7b4793a640 1 librados: shutdown 2024-03-25T18:10:05.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:12.405 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:12.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.521+0000 7f74473f6640 10 monclient: get_monmap_and_config 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.521+0000 7f74473f6640 10 monclient: build_initial_monmap 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.521+0000 7f74473f6640 10 monclient: monmap: 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:12.522258+0000 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:12.522258+0000 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:12.522 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.523 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.522+0000 7f74473f6640 1 Processor -- start 2024-03-25T18:10:12.523 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.522+0000 7f74473f6640 1 -- start start 2024-03-25T18:10:12.523 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.522+0000 7f74473f6640 10 monclient: init 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-b con 0x7f74400a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-c con 0x7f744013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-a con 0x7f74401430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7440142520 con 0x7f74401430c0 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400b28a0 con 0x7f744013f710 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400b2a30 con 0x7f74400a98f0 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:12.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.523+0000 7f74473f6640 10 monclient(hunting): authenticate will time out at 7321.785117s 2024-03-25T18:10:12.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.534+0000 7f7445993640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f74401430c0 0x7f744014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39516/0 (socket says 172.21.15.46:39516) 2024-03-25T18:10:12.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.534+0000 7f7445993640 1 -- 172.21.15.46:0/1596968786 learned_addr learned my addr 172.21.15.46:0/1596968786 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:12.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417049441 0 0) 0x7f74400b28a0 con 0x7f744013f710 2024-03-25T18:10:12.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:10:12.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.553+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f742c003520 con 0x7f744013f710 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679595150 0 0) 0x7f74400b2a30 con 0x7f74400a98f0 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:10:12.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.554+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74400b28a0 con 0x7f74400a98f0 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506320738 0 0) 0x7f7440142520 con 0x7f74401430c0 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:10:12.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.558+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74400b2a30 con 0x7f74401430c0 2024-03-25T18:10:12.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458774822 0 0) 0x7f74400b2a30 con 0x7f74401430c0 2024-03-25T18:10:12.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 >> v1:172.21.15.46:6790/0 conn(0x7f74400a98f0 legacy=0x7f74400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 >> v1:172.21.15.164:6789/0 conn(0x7f744013f710 legacy=0x7f74401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 monclient: found mon.noname-a 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400b44c0 con 0x7f74401430c0 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f74473f6640 5 monclient: authenticate success, global_id 4281 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f74473f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7438001870 con 0x7f74401430c0 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.562+0000 7f7437fff640 10 monclient: dump: 2024-03-25T18:10:12.564 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.563+0000 7f74473f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:12.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7438001b40 con 0x7f74401430c0 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 1 -- 172.21.15.46:0/1596968786 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7438001ed0 con 0x7f74401430c0 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f7437fff640 10 monclient: dump: 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f74473f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:12.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.574+0000 7f74473f6640 4 set_mon_vals no callback set 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 10 monclient: shutdown 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 1 -- 172.21.15.46:0/1596968786 >> v1:172.21.15.46:6789/0 conn(0x7f74401430c0 legacy=0x7f744014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 1 -- 172.21.15.46:0/1596968786 shutdown_connections 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 1 -- 172.21.15.46:0/1596968786 >> 172.21.15.46:0/1596968786 conn(0x7f74400b7ea0 msgr2=0x7f74400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 1 -- 172.21.15.46:0/1596968786 shutdown_connections 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.575+0000 7f74473f6640 1 -- 172.21.15.46:0/1596968786 wait complete. 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.576+0000 7f74473f6640 10 monclient: build_initial_monmap 2024-03-25T18:10:12.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.576+0000 7f74473f6640 10 monclient: monmap: 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.576+0000 7f74473f6640 1 librados: starting msgr at 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.576+0000 7f74473f6640 1 librados: starting objecter 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.576+0000 7f74473f6640 1 Processor -- start 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 -- start start 2024-03-25T18:10:12.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 librados: setting wanted keys 2024-03-25T18:10:12.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 librados: calling monclient init 2024-03-25T18:10:12.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient: init 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-b con 0x7f74400a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-a con 0x7f744013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient(hunting): picked mon.noname-c con 0x7f74401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f744013b380 con 0x7f744013f710 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7440139dc0 con 0x7f74400a98f0 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f744012fab0 con 0x7f74401430c0 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.577+0000 7f74473f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:12.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.578+0000 7f74473f6640 10 monclient(hunting): authenticate will time out at 7321.839541s 2024-03-25T18:10:12.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.593+0000 7f7445192640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f74400a98f0 0x7f744012f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45294/0 (socket says 172.21.15.46:45294) 2024-03-25T18:10:12.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.593+0000 7f7445192640 1 -- 172.21.15.46:0/3274671717 learned_addr learned my addr 172.21.15.46:0/3274671717 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:12.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.594+0000 7f7435ffb640 10 client.?.objecter ms_handle_connect 0x7f74400a98f0 2024-03-25T18:10:12.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.594+0000 7f7435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.595+0000 7f7435ffb640 10 client.?.objecter ms_handle_connect 0x7f74401430c0 2024-03-25T18:10:12.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.595+0000 7f7435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71711755 0 0) 0x7f7440139dc0 con 0x7f74400a98f0 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:10:12.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.600+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7424004eb0 con 0x7f74400a98f0 2024-03-25T18:10:12.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.603+0000 7f7435ffb640 10 client.?.objecter ms_handle_connect 0x7f744013f710 2024-03-25T18:10:12.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.603+0000 7f7435ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459478502 0 0) 0x7f7424004eb0 con 0x7f74400a98f0 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7440139dc0 con 0x7f74400a98f0 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f743c001990 con 0x7f74400a98f0 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.609 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956733947 0 0) 0x7f744012fab0 con 0x7f74401430c0 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:10:12.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.608+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7424004eb0 con 0x7f74401430c0 2024-03-25T18:10:12.615 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584390941 0 0) 0x7f744013b380 con 0x7f744013f710 2024-03-25T18:10:12.615 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.615 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.615 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:10:12.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.615+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f744012fab0 con 0x7f744013f710 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.616+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080104702 0 0) 0x7f7440139dc0 con 0x7f74400a98f0 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 >> v1:172.21.15.46:6790/0 conn(0x7f74401430c0 legacy=0x7f74401e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 >> v1:172.21.15.46:6789/0 conn(0x7f744013f710 legacy=0x7f74401df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 1 monclient: found mon.b 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744013a2e0 con 0x7f74400a98f0 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f7435ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 10 monclient: _renew_subs 2024-03-25T18:10:12.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74400b6cc0 con 0x7f74400a98f0 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 10 monclient: _renew_subs 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7440139860 con 0x7f74400a98f0 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.617+0000 7f74473f6640 1 librados: init done 2024-03-25T18:10:12.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.618+0000 7f74473f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.618+0000 7f74473f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.618+0000 7f74473f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:12.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.618+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440139f50 con 0x7f74400a98f0 2024-03-25T18:10:12.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f743c0035f0 con 0x7f74400a98f0 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f743c003a40 con 0x7f74400a98f0 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f7435ffb640 10 monclient: dump: 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:12.913 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:12.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.912+0000 7f74377fe640 4 set_mon_vals no callback set 2024-03-25T18:10:12.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.913+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f743c003c60 con 0x7f74400a98f0 2024-03-25T18:10:12.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.914+0000 7f7444991640 10 monclient: get_auth_request con 0x7f742406de80 auth_method 0 2024-03-25T18:10:12.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.914+0000 7f7435ffb640 10 client.4288.objecter ms_handle_connect 0x7f742406de80 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.920+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f743c054770 con 0x7f74400a98f0 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.920+0000 7f7435ffb640 10 client.4288.objecter ms_dispatch 0x7f7440000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.920+0000 7f7435ffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.920+0000 7f7435ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.920+0000 7f7435ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.921+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f743c050280 con 0x7f74400a98f0 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.921+0000 7f7435ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:12.921+0000 7f7435ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:13.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.167+0000 7f74473f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.167+0000 7f74473f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.167+0000 7f74473f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:13.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.167+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f744014f2d0 con 0x7f74400a98f0 2024-03-25T18:10:13.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.196+0000 7f7435ffb640 1 -- 172.21.15.46:0/3274671717 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f743c0521b0 con 0x7f74400a98f0 2024-03-25T18:10:13.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.196+0000 7f7435ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.196+0000 7f7435ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:13.242 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:13.242 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 10 librados: watch_flush enter 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 10 librados: watch_flush exit 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 >> v1:172.21.15.164:6800/1612084612 conn(0x7f742406de80 legacy=0x7f7424070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 10 monclient: shutdown 2024-03-25T18:10:13.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 >> v1:172.21.15.164:6789/0 conn(0x7f74400a98f0 legacy=0x7f744012f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 shutdown_connections 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 >> 172.21.15.46:0/3274671717 conn(0x7f74400b7ea0 msgr2=0x7f7440074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 shutdown_connections 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 -- 172.21.15.46:0/3274671717 wait complete. 2024-03-25T18:10:13.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:13.199+0000 7f74473f6640 1 librados: shutdown 2024-03-25T18:10:13.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:18.657 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:54.575 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.674 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:54.592 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.710 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:01:54.628 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.732 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:54.650 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.743 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:54.615 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.744 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:54.662 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.747 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:54.619 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.780 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:01:54.651 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:18.784 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:01:54.701 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:20.051 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:20.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.164+0000 7f9daefc8640 10 monclient: get_monmap_and_config 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.164+0000 7f9daefc8640 10 monclient: build_initial_monmap 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.164+0000 7f9daefc8640 10 monclient: monmap: 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:20.165181+0000 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:20.165181+0000 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:20.165 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 1 Processor -- start 2024-03-25T18:10:20.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 1 -- start start 2024-03-25T18:10:20.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient: init 2024-03-25T18:10:20.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9da80a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9da813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9da81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da8142520 con 0x7f9da81430c0 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da80b28a0 con 0x7f9da813f710 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da80b2a30 con 0x7f9da80a98f0 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:20.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.165+0000 7f9daefc8640 10 monclient(hunting): authenticate will time out at 7329.427523s 2024-03-25T18:10:20.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.170+0000 7f9dad565640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9da81430c0 0x7f9da814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57998/0 (socket says 172.21.15.46:57998) 2024-03-25T18:10:20.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.170+0000 7f9dad565640 1 -- 172.21.15.46:0/4246453813 learned_addr learned my addr 172.21.15.46:0/4246453813 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:20.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817089499 0 0) 0x7f9da8142520 con 0x7f9da81430c0 2024-03-25T18:10:20.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:10:20.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.182+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003520 con 0x7f9da81430c0 2024-03-25T18:10:20.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660654926 0 0) 0x7f9da80b2a30 con 0x7f9da80a98f0 2024-03-25T18:10:20.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:10:20.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.189+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da8142520 con 0x7f9da80a98f0 2024-03-25T18:10:20.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740169213 0 0) 0x7f9da80b28a0 con 0x7f9da813f710 2024-03-25T18:10:20.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:10:20.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.194+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da80b2a30 con 0x7f9da813f710 2024-03-25T18:10:20.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694939370 0 0) 0x7f9d98003520 con 0x7f9da81430c0 2024-03-25T18:10:20.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 >> v1:172.21.15.46:6790/0 conn(0x7f9da80a98f0 legacy=0x7f9da80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 >> v1:172.21.15.164:6789/0 conn(0x7f9da813f710 legacy=0x7f9da81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 monclient: found mon.noname-a 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80b44c0 con 0x7f9da81430c0 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9da0001a10 con 0x7f9da81430c0 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.209+0000 7f9da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.210+0000 7f9da77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.210+0000 7f9da77fe640 10 monclient: dump: 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.211 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.210+0000 7f9daefc8640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:10:20.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.210+0000 7f9daefc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:20.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9da0001b70 con 0x7f9da81430c0 2024-03-25T18:10:20.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:20.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 1 -- 172.21.15.46:0/4246453813 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9da0001f80 con 0x7f9da81430c0 2024-03-25T18:10:20.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9da77fe640 10 monclient: dump: 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9daefc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:20.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.235+0000 7f9daefc8640 4 set_mon_vals no callback set 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 10 monclient: shutdown 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 1 -- 172.21.15.46:0/4246453813 >> v1:172.21.15.46:6789/0 conn(0x7f9da81430c0 legacy=0x7f9da814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 1 -- 172.21.15.46:0/4246453813 shutdown_connections 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 1 -- 172.21.15.46:0/4246453813 >> 172.21.15.46:0/4246453813 conn(0x7f9da80b7ea0 msgr2=0x7f9da80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 1 -- 172.21.15.46:0/4246453813 shutdown_connections 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.236+0000 7f9daefc8640 1 -- 172.21.15.46:0/4246453813 wait complete. 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 10 monclient: build_initial_monmap 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 10 monclient: monmap: 2024-03-25T18:10:20.238 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 librados: starting msgr at 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 librados: starting objecter 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 Processor -- start 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 -- start start 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 librados: setting wanted keys 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 1 librados: calling monclient init 2024-03-25T18:10:20.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.237+0000 7f9daefc8640 10 monclient: init 2024-03-25T18:10:20.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:20.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9da813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9da81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da8136c50 con 0x7f9da80a98f0 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da8136ec0 con 0x7f9da81430c0 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da81317f0 con 0x7f9da813f710 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:20.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.238+0000 7f9daefc8640 10 monclient(hunting): authenticate will time out at 7329.500017s 2024-03-25T18:10:20.245 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.245+0000 7f9dad565640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9da81430c0 0x7f9da81db380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:37022/0 (socket says 172.21.15.46:37022) 2024-03-25T18:10:20.245 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.245+0000 7f9dad565640 1 -- 172.21.15.46:0/3201744599 learned_addr learned my addr 172.21.15.46:0/3201744599 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:20.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.247+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da81430c0 2024-03-25T18:10:20.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.247+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.251+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da80a98f0 2024-03-25T18:10:20.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.251+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.251+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da813f710 2024-03-25T18:10:20.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.251+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298286903 0 0) 0x7f9da8136ec0 con 0x7f9da81430c0 2024-03-25T18:10:20.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:10:20.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.260+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c004eb0 con 0x7f9da81430c0 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232488210 0 0) 0x7f9da81317f0 con 0x7f9da813f710 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:10:20.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da8136ec0 con 0x7f9da813f710 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004714695 0 0) 0x7f9da8136c50 con 0x7f9da80a98f0 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:10:20.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.264+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da81317f0 con 0x7f9da80a98f0 2024-03-25T18:10:20.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293237258 0 0) 0x7f9d8c004eb0 con 0x7f9da81430c0 2024-03-25T18:10:20.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9da8136c50 con 0x7f9da81430c0 2024-03-25T18:10:20.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9da0001ec0 con 0x7f9da81430c0 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 10 monclient(hunting): dump: 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.278+0000 7f9da57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763009082 0 0) 0x7f9da8136ec0 con 0x7f9da813f710 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c004eb0 con 0x7f9da813f710 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9d94001870 con 0x7f9da813f710 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.279+0000 7f9da57fa640 10 monclient(hunting): dump: 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:20.280 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:20.281 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471972792 0 0) 0x7f9da8136c50 con 0x7f9da81430c0 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 >> v1:172.21.15.46:6790/0 conn(0x7f9da813f710 legacy=0x7f9da81dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 >> v1:172.21.15.46:6789/0 conn(0x7f9da80a98f0 legacy=0x7f9da81310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 1 monclient: found mon.b 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8131980 con 0x7f9da81430c0 2024-03-25T18:10:20.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 10 monclient: _finish_auth 0 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9da57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 10 monclient: _renew_subs 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da8135070 con 0x7f9da81430c0 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 10 monclient: _renew_subs 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da80b7870 con 0x7f9da81430c0 2024-03-25T18:10:20.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.287+0000 7f9daefc8640 1 librados: init done 2024-03-25T18:10:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.288+0000 7f9daefc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.289+0000 7f9daefc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.289+0000 7f9daefc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.289+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da80627b0 con 0x7f9da81430c0 2024-03-25T18:10:20.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.517+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9da0004e10 con 0x7f9da81430c0 2024-03-25T18:10:20.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.517+0000 7f9da57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:20.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.517+0000 7f9da6ffd640 4 set_mon_vals no callback set 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.557+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9da0002100 con 0x7f9da81430c0 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.557+0000 7f9da57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.557+0000 7f9da57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.557+0000 7f9da57fa640 10 monclient: dump: 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:20.558 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.558+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f9da001b010 con 0x7f9da81430c0 2024-03-25T18:10:20.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.559+0000 7f9dacd64640 10 monclient: get_auth_request con 0x7f9d8c06df70 auth_method 0 2024-03-25T18:10:20.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.559+0000 7f9da57fa640 10 client.4294.objecter ms_handle_connect 0x7f9d8c06df70 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.564+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f9da0054d30 con 0x7f9da81430c0 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.564+0000 7f9da57fa640 10 client.4294.objecter ms_dispatch 0x7f9da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.564+0000 7f9da57fa640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.564+0000 7f9da57fa640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.564+0000 7f9da57fa640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.565+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9da0050670 con 0x7f9da81430c0 2024-03-25T18:10:20.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.565+0000 7f9da57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.565+0000 7f9da57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.803+0000 7f9daefc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.803+0000 7f9daefc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.803+0000 7f9daefc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.803+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9da80b7ac0 con 0x7f9da81430c0 2024-03-25T18:10:20.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.856+0000 7f9da57fa640 1 -- 172.21.15.46:0/3201744599 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f9da0052770 con 0x7f9da81430c0 2024-03-25T18:10:20.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.856+0000 7f9da57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.856+0000 7f9da57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:20.857 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:20.857 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 10 librados: watch_flush enter 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 10 librados: watch_flush exit 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 >> v1:172.21.15.164:6800/1612084612 conn(0x7f9d8c06df70 legacy=0x7f9d8c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 10 monclient: shutdown 2024-03-25T18:10:20.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 >> v1:172.21.15.164:6789/0 conn(0x7f9da81430c0 legacy=0x7f9da81db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 shutdown_connections 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 >> 172.21.15.46:0/3201744599 conn(0x7f9da80b7ea0 msgr2=0x7f9da813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 shutdown_connections 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 -- 172.21.15.46:0/3201744599 wait complete. 2024-03-25T18:10:20.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:20.858+0000 7f9daefc8640 1 librados: shutdown 2024-03-25T18:10:20.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:27.677 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 10 monclient: get_monmap_and_config 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 10 monclient: build_initial_monmap 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 10 monclient: monmap: 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:27.789804+0000 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:27.789804+0000 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:27.790 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 1 Processor -- start 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 1 -- start start 2024-03-25T18:10:27.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.788+0000 7f2f3550e640 10 monclient: init 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f301430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f30142520 con 0x7f2f301430c0 2024-03-25T18:10:27.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300b28a0 con 0x7f2f3013f710 2024-03-25T18:10:27.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300b2a30 con 0x7f2f300a98f0 2024-03-25T18:10:27.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.789+0000 7f2f3550e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:27.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.790+0000 7f2f3550e640 10 monclient(hunting): authenticate will time out at 7337.052548s 2024-03-25T18:10:27.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.800+0000 7f2f2e7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2f3013f710 0x7f2f301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34464/0 (socket says 172.21.15.46:34464) 2024-03-25T18:10:27.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.800+0000 7f2f2e7fc640 1 -- 172.21.15.46:0/3898469217 learned_addr learned my addr 172.21.15.46:0/3898469217 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437621908 0 0) 0x7f2f300b2a30 con 0x7f2f300a98f0 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:10:27.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.813+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f18003520 con 0x7f2f300a98f0 2024-03-25T18:10:27.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373262704 0 0) 0x7f2f300b28a0 con 0x7f2f3013f710 2024-03-25T18:10:27.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:10:27.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.824+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f300b2a30 con 0x7f2f3013f710 2024-03-25T18:10:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2717218325 0 0) 0x7f2f18003520 con 0x7f2f300a98f0 2024-03-25T18:10:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 >> v1:172.21.15.164:6789/0 conn(0x7f2f3013f710 legacy=0x7f2f301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 >> v1:172.21.15.46:6789/0 conn(0x7f2f301430c0 legacy=0x7f2f3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 monclient: found mon.noname-b 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300b44c0 con 0x7f2f300a98f0 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f20001690 con 0x7f2f300a98f0 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f2dffb640 10 monclient: dump: 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.829 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f3550e640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:10:27.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.826+0000 7f2f3550e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.837+0000 7f2f2dffb640 1 -- 172.21.15.46:0/3898469217 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f200017f0 con 0x7f2f300a98f0 2024-03-25T18:10:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.837+0000 7f2f2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.837+0000 7f2f3550e640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.837+0000 7f2f3550e640 4 set_mon_vals no callback set 2024-03-25T18:10:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 10 monclient: shutdown 2024-03-25T18:10:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 1 -- 172.21.15.46:0/3898469217 >> v1:172.21.15.46:6790/0 conn(0x7f2f300a98f0 legacy=0x7f2f300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 1 -- 172.21.15.46:0/3898469217 shutdown_connections 2024-03-25T18:10:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 1 -- 172.21.15.46:0/3898469217 >> 172.21.15.46:0/3898469217 conn(0x7f2f300b7ea0 msgr2=0x7f2f300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 1 -- 172.21.15.46:0/3898469217 shutdown_connections 2024-03-25T18:10:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.838+0000 7f2f3550e640 1 -- 172.21.15.46:0/3898469217 wait complete. 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 10 monclient: build_initial_monmap 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 10 monclient: monmap: 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:27.841 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 librados: starting msgr at 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 librados: starting objecter 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 Processor -- start 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 -- start start 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 librados: setting wanted keys 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 1 librados: calling monclient init 2024-03-25T18:10:27.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.839+0000 7f2f3550e640 10 monclient: init 2024-03-25T18:10:27.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:27.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:27.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f301430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f30139680 con 0x7f2f301430c0 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f30138cf0 con 0x7f2f300a98f0 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f30136620 con 0x7f2f3013f710 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:27.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.840+0000 7f2f3550e640 10 monclient(hunting): authenticate will time out at 7337.103394s 2024-03-25T18:10:27.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.849+0000 7f2f2effd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2f300a98f0 0x7f2f301371a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34508/0 (socket says 172.21.15.46:34508) 2024-03-25T18:10:27.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.849+0000 7f2f2effd640 1 -- 172.21.15.46:0/1099270990 learned_addr learned my addr 172.21.15.46:0/1099270990 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:27.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.855+0000 7f2f177fe640 10 client.?.objecter ms_handle_connect 0x7f2f301430c0 2024-03-25T18:10:27.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.855+0000 7f2f177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.856+0000 7f2f177fe640 10 client.?.objecter ms_handle_connect 0x7f2f300a98f0 2024-03-25T18:10:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.856+0000 7f2f177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940536749 0 0) 0x7f2f30138cf0 con 0x7f2f300a98f0 2024-03-25T18:10:27.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:10:27.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.863+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c004eb0 con 0x7f2f300a98f0 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220824168 0 0) 0x7f2f30139680 con 0x7f2f301430c0 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:10:27.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.867+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f30138cf0 con 0x7f2f301430c0 2024-03-25T18:10:27.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.873+0000 7f2f177fe640 10 client.?.objecter ms_handle_connect 0x7f2f3013f710 2024-03-25T18:10:27.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.873+0000 7f2f177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806060852 0 0) 0x7f2f30138cf0 con 0x7f2f301430c0 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f30139680 con 0x7f2f301430c0 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f28001b00 con 0x7f2f301430c0 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 10 monclient(hunting): dump: 2024-03-25T18:10:27.887 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:27.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.885+0000 7f2f177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712648396 0 0) 0x7f2f1c004eb0 con 0x7f2f300a98f0 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f30138cf0 con 0x7f2f300a98f0 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f200035f0 con 0x7f2f300a98f0 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.889+0000 7f2f177fe640 10 monclient(hunting): dump: 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.891 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:27.892 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:27.892 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:27.892 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:27.892 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:27.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687825223 0 0) 0x7f2f30136620 con 0x7f2f3013f710 2024-03-25T18:10:27.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:10:27.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.892+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c004eb0 con 0x7f2f3013f710 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 797034032 0 0) 0x7f2f30139680 con 0x7f2f301430c0 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 >> v1:172.21.15.46:6790/0 conn(0x7f2f3013f710 legacy=0x7f2f301dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 >> v1:172.21.15.164:6789/0 conn(0x7f2f300a98f0 legacy=0x7f2f301371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 1 monclient: found mon.a 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300b1620 con 0x7f2f301430c0 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:27.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 5 monclient: authenticate success, global_id 4296 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 10 monclient: _renew_subs 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f30139110 con 0x7f2f301430c0 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 10 monclient: _renew_subs 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3013b770 con 0x7f2f301430c0 2024-03-25T18:10:27.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.894+0000 7f2f3550e640 1 librados: init done 2024-03-25T18:10:27.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.895+0000 7f2f3550e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:27.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.895+0000 7f2f3550e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:27.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.895+0000 7f2f3550e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:27.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:27.895+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f30139460 con 0x7f2f301430c0 2024-03-25T18:10:28.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f28001ff0 con 0x7f2f301430c0 2024-03-25T18:10:28.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:28.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f28003b10 con 0x7f2f301430c0 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f177fe640 10 monclient: dump: 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:28.215 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:28.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.212+0000 7f2f2d7fa640 4 set_mon_vals no callback set 2024-03-25T18:10:28.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.213+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f2f28003cf0 con 0x7f2f301430c0 2024-03-25T18:10:28.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.214+0000 7f2f2effd640 10 monclient: get_auth_request con 0x7f2f1c06df50 auth_method 0 2024-03-25T18:10:28.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.214+0000 7f2f177fe640 10 client.4296.objecter ms_handle_connect 0x7f2f1c06df50 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f2f280537a0 con 0x7f2f301430c0 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 10 client.4296.objecter ms_dispatch 0x7f2f30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f2804f2b0 con 0x7f2f301430c0 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:28.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.220+0000 7f2f177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:28.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.476+0000 7f2f3550e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.476+0000 7f2f3550e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.476+0000 7f2f3550e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:28.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.476+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f30139830 con 0x7f2f301430c0 2024-03-25T18:10:28.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.574+0000 7f2f177fe640 1 -- 172.21.15.46:0/1099270990 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f2f30139830 con 0x7f2f301430c0 2024-03-25T18:10:28.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.574+0000 7f2f177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.574+0000 7f2f177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:28.577 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:28.577 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.576+0000 7f2f3550e640 10 librados: watch_flush enter 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 10 librados: watch_flush exit 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 10 client.4296.objecter successfully canceled tick 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 >> v1:172.21.15.164:6800/1612084612 conn(0x7f2f1c06df50 legacy=0x7f2f1c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 10 monclient: shutdown 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 >> v1:172.21.15.46:6789/0 conn(0x7f2f301430c0 legacy=0x7f2f301e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:28.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 shutdown_connections 2024-03-25T18:10:28.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 >> 172.21.15.46:0/1099270990 conn(0x7f2f300b7ea0 msgr2=0x7f2f30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:28.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 shutdown_connections 2024-03-25T18:10:28.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 -- 172.21.15.46:0/1099270990 wait complete. 2024-03-25T18:10:28.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:28.577+0000 7f2f3550e640 1 librados: shutdown 2024-03-25T18:10:28.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:35.397 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:35.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 10 monclient: get_monmap_and_config 2024-03-25T18:10:35.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 10 monclient: build_initial_monmap 2024-03-25T18:10:35.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 10 monclient: monmap: 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:35.516825+0000 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:35.516825+0000 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:35.517 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 1 Processor -- start 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 1 -- start start 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.515+0000 7f37b8ac1640 10 monclient: init 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.516+0000 7f37b8ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.516+0000 7f37b8ac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.516+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f37b40705b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.516+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f37b41402d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f37b4149ae0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40718c0 con 0x7f37b4149ae0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40714c0 con 0x7f37b41402d0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b4072b50 con 0x7f37b40705b0 2024-03-25T18:10:35.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:35.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.517+0000 7f37b8ac1640 10 monclient(hunting): authenticate will time out at 7344.779839s 2024-03-25T18:10:35.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.526+0000 7f37b1d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f37b41402d0 0x7f37b4140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53096/0 (socket says 172.21.15.46:53096) 2024-03-25T18:10:35.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.526+0000 7f37b1d9b640 1 -- 172.21.15.46:0/4237957745 learned_addr learned my addr 172.21.15.46:0/4237957745 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538495456 0 0) 0x7f37b40714c0 con 0x7f37b41402d0 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:10:35.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.536+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c003520 con 0x7f37b41402d0 2024-03-25T18:10:35.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385592810 0 0) 0x7f37b40718c0 con 0x7f37b4149ae0 2024-03-25T18:10:35.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:10:35.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.555+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b40714c0 con 0x7f37b4149ae0 2024-03-25T18:10:35.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795168102 0 0) 0x7f37b4072b50 con 0x7f37b40705b0 2024-03-25T18:10:35.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:10:35.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.560+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b40718c0 con 0x7f37b40705b0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279922380 0 0) 0x7f379c003520 con 0x7f37b41402d0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 >> v1:172.21.15.46:6790/0 conn(0x7f37b40705b0 legacy=0x7f37b413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 >> v1:172.21.15.46:6789/0 conn(0x7f37b4149ae0 legacy=0x7f37b414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: _finish_hunting 0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 monclient: found mon.noname-c 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b413dd10 con 0x7f37b41402d0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: _finish_auth 0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: _check_auth_tickets 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37a8001690 con 0x7f37b41402d0 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b159a640 10 monclient: dump: 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b8ac1640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:10:35.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.574+0000 7f37b8ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:35.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a80017f0 con 0x7f37b41402d0 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 1 -- 172.21.15.46:0/4237957745 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37a8001c00 con 0x7f37b41402d0 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b159a640 10 monclient: dump: 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.592 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.593 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.593 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b8ac1640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:35.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.590+0000 7f37b8ac1640 4 set_mon_vals no callback set 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 10 monclient: shutdown 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 1 -- 172.21.15.46:0/4237957745 >> v1:172.21.15.164:6789/0 conn(0x7f37b41402d0 legacy=0x7f37b4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 1 -- 172.21.15.46:0/4237957745 shutdown_connections 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 1 -- 172.21.15.46:0/4237957745 >> 172.21.15.46:0/4237957745 conn(0x7f37b40b5d10 msgr2=0x7f37b40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 1 -- 172.21.15.46:0/4237957745 shutdown_connections 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.591+0000 7f37b8ac1640 1 -- 172.21.15.46:0/4237957745 wait complete. 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.592+0000 7f37b8ac1640 10 monclient: build_initial_monmap 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.592+0000 7f37b8ac1640 10 monclient: monmap: 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:35.594 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:35.595 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 librados: starting msgr at 2024-03-25T18:10:35.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 librados: starting objecter 2024-03-25T18:10:35.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 Processor -- start 2024-03-25T18:10:35.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 -- start start 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 librados: setting wanted keys 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 librados: calling monclient init 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient: init 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f37b40705b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f37b41402d0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f37b4149ae0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40ac240 con 0x7f37b4149ae0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b41dae50 con 0x7f37b40705b0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b41dba80 con 0x7f37b41402d0 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:35.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.593+0000 7f37b8ac1640 10 monclient(hunting): authenticate will time out at 7344.856416s 2024-03-25T18:10:35.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.599+0000 7f37b259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f37b40705b0 0x7f37b40aabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53180/0 (socket says 172.21.15.46:53180) 2024-03-25T18:10:35.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.599+0000 7f37b259c640 1 -- 172.21.15.46:0/2250569150 learned_addr learned my addr 172.21.15.46:0/2250569150 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:35.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.604+0000 7f37a2ffd640 10 client.?.objecter ms_handle_connect 0x7f37b40705b0 2024-03-25T18:10:35.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.604+0000 7f37a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.615+0000 7f37a2ffd640 10 client.?.objecter ms_handle_connect 0x7f37b41402d0 2024-03-25T18:10:35.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.615+0000 7f37a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.615+0000 7f37a2ffd640 10 client.?.objecter ms_handle_connect 0x7f37b4149ae0 2024-03-25T18:10:35.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.615+0000 7f37a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253506088 0 0) 0x7f37b41dae50 con 0x7f37b40705b0 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:10:35.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.617+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794004eb0 con 0x7f37b40705b0 2024-03-25T18:10:35.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952108068 0 0) 0x7f37b40ac240 con 0x7f37b4149ae0 2024-03-25T18:10:35.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:10:35.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.619+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b41dae50 con 0x7f37b4149ae0 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256762388 0 0) 0x7f37b41dba80 con 0x7f37b41402d0 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:10:35.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.625+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b40ac240 con 0x7f37b41402d0 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075455024 0 0) 0x7f37b41dae50 con 0x7f37b4149ae0 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37b41dba80 con 0x7f37b4149ae0 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37ac001b50 con 0x7f37b4149ae0 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.632 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.633 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.630+0000 7f37a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:35.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668519976 0 0) 0x7f3794004eb0 con 0x7f37b40705b0 2024-03-25T18:10:35.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37b41dae50 con 0x7f37b40705b0 2024-03-25T18:10:35.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37a4001a10 con 0x7f37b40705b0 2024-03-25T18:10:35.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.634+0000 7f37a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.636 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.637+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 156337790 0 0) 0x7f37b40ac240 con 0x7f37b41402d0 2024-03-25T18:10:35.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.637+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3794004eb0 con 0x7f37b41402d0 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.639+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37a8001b00 con 0x7f37b41402d0 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.639+0000 7f37a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.639+0000 7f37a2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.639+0000 7f37a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.641 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.642 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3235007223 0 0) 0x7f37b41dba80 con 0x7f37b4149ae0 2024-03-25T18:10:35.646 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 >> v1:172.21.15.46:6790/0 conn(0x7f37b41402d0 legacy=0x7f37b41db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 >> v1:172.21.15.164:6789/0 conn(0x7f37b40705b0 legacy=0x7f37b40aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 1 monclient: found mon.a 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b41dbc10 con 0x7f37b4149ae0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37b8ac1640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37b8ac1640 10 monclient: _renew_subs 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b41422e0 con 0x7f37b4149ae0 2024-03-25T18:10:35.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.644+0000 7f37b8ac1640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.645+0000 7f37b8ac1640 10 monclient: _renew_subs 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.645+0000 7f37b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.645+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b40aa030 con 0x7f37b4149ae0 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.645+0000 7f37b8ac1640 1 librados: init done 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.646+0000 7f37b8ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.646+0000 7f37b8ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.646+0000 7f37b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:35.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.646+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b41db060 con 0x7f37b4149ae0 2024-03-25T18:10:35.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.966+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37ac002080 con 0x7f37b4149ae0 2024-03-25T18:10:35.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.966+0000 7f37a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:35.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.966+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f37ac003b10 con 0x7f37b4149ae0 2024-03-25T18:10:35.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.966+0000 7f37a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.967+0000 7f37a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.967+0000 7f37a2ffd640 10 monclient: dump: 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:35.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.967+0000 7f37b0d99640 4 set_mon_vals no callback set 2024-03-25T18:10:35.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.968+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f37ac019b20 con 0x7f37b4149ae0 2024-03-25T18:10:35.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.968+0000 7f37b259c640 10 monclient: get_auth_request con 0x7f379406dff0 auth_method 0 2024-03-25T18:10:35.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.969+0000 7f37a2ffd640 10 client.4302.objecter ms_handle_connect 0x7f379406dff0 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f37ac053810 con 0x7f37b4149ae0 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 10 client.4302.objecter ms_dispatch 0x7f37b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37ac04f150 con 0x7f37b4149ae0 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:35.974+0000 7f37a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:36.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.195+0000 7f37b8ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:36.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.195+0000 7f37b8ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:36.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.195+0000 7f37b8ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:10:36.197 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.195+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37b40ac040 con 0x7f37b4149ae0 2024-03-25T18:10:36.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.271+0000 7f37a2ffd640 1 -- 172.21.15.46:0/2250569150 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f37ac051250 con 0x7f37b4149ae0 2024-03-25T18:10:36.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.271+0000 7f37a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:36.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.271+0000 7f37a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:36.273 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:36.273 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:36.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 10 librados: watch_flush enter 2024-03-25T18:10:36.275 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 10 librados: watch_flush exit 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 >> v1:172.21.15.164:6800/1612084612 conn(0x7f379406dff0 legacy=0x7f37940704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 10 monclient: shutdown 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.273+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 >> v1:172.21.15.46:6789/0 conn(0x7f37b4149ae0 legacy=0x7f37b4139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.274+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 shutdown_connections 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.274+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 >> 172.21.15.46:0/2250569150 conn(0x7f37b40b5d10 msgr2=0x7f37b413c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.274+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 shutdown_connections 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.274+0000 7f37b8ac1640 1 -- 172.21.15.46:0/2250569150 wait complete. 2024-03-25T18:10:36.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:36.274+0000 7f37b8ac1640 1 librados: shutdown 2024-03-25T18:10:36.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:38.716 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:14.634 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.733 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:14.651 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.779 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:14.650 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.784 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:14.702 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.785 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:14.656 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.789 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:14.707 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.801 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:14.719 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.812 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:14.683 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:38.834 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:14.752 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:43.095 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.209+0000 7fa297472640 10 monclient: get_monmap_and_config 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.209+0000 7fa297472640 10 monclient: build_initial_monmap 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.209+0000 7fa297472640 10 monclient: monmap: 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:43.211019+0000 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:43.211019+0000 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:43.211 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:43.212 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 1 Processor -- start 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 1 -- start start 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient: init 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient(hunting): picked mon.noname-b con 0x7fa2900a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient(hunting): picked mon.noname-c con 0x7fa29013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 10 monclient(hunting): picked mon.noname-a con 0x7fa2901430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.210+0000 7fa297472640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa290142520 con 0x7fa2901430c0 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.211+0000 7fa297472640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2900b28a0 con 0x7fa29013f710 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.211+0000 7fa297472640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2900b2a30 con 0x7fa2900a98f0 2024-03-25T18:10:43.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.211+0000 7fa297472640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:43.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.211+0000 7fa297472640 10 monclient(hunting): authenticate will time out at 7352.473565s 2024-03-25T18:10:43.221 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.219+0000 7fa29520e640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fa2900a98f0 0x7fa2900a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:57226/0 (socket says 172.21.15.46:57226) 2024-03-25T18:10:43.221 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.219+0000 7fa29520e640 1 -- 172.21.15.46:0/363935224 learned_addr learned my addr 172.21.15.46:0/363935224 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941447697 0 0) 0x7fa2900b2a30 con 0x7fa2900a98f0 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:10:43.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.238+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa2900a98f0 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672428393 0 0) 0x7fa2900b28a0 con 0x7fa29013f710 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:10:43.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.242+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2900b2a30 con 0x7fa29013f710 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261189312 0 0) 0x7fa290142520 con 0x7fa2901430c0 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:10:43.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.250+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2900b28a0 con 0x7fa2901430c0 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928570707 0 0) 0x7fa27c003520 con 0x7fa2900a98f0 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 >> v1:172.21.15.164:6789/0 conn(0x7fa29013f710 legacy=0x7fa2901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 >> v1:172.21.15.46:6789/0 conn(0x7fa2901430c0 legacy=0x7fa29014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 monclient: found mon.noname-b 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:43.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900b44c0 con 0x7fa2900a98f0 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa28c001690 con 0x7fa2900a98f0 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa287fff640 10 monclient: dump: 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.256 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:43.257 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:43.257 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:43.257 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa297472640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:10:43.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.253+0000 7fa297472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa28c0017f0 con 0x7fa2900a98f0 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 1 -- 172.21.15.46:0/363935224 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa28c001c00 con 0x7fa2900a98f0 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa297472640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa297472640 4 set_mon_vals no callback set 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:43.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.276+0000 7fa287fff640 10 monclient: dump: 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:43.279 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 10 monclient: shutdown 2024-03-25T18:10:43.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:43.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 1 -- 172.21.15.46:0/363935224 >> v1:172.21.15.46:6790/0 conn(0x7fa2900a98f0 legacy=0x7fa2900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 1 -- 172.21.15.46:0/363935224 shutdown_connections 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 1 -- 172.21.15.46:0/363935224 >> 172.21.15.46:0/363935224 conn(0x7fa2900b7ea0 msgr2=0x7fa2900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 1 -- 172.21.15.46:0/363935224 shutdown_connections 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.277+0000 7fa297472640 1 -- 172.21.15.46:0/363935224 wait complete. 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.278+0000 7fa297472640 10 monclient: build_initial_monmap 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.278+0000 7fa297472640 10 monclient: monmap: 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:43.282 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.278+0000 7fa297472640 1 librados: starting msgr at 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.278+0000 7fa297472640 1 librados: starting objecter 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.278+0000 7fa297472640 1 Processor -- start 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 -- start start 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 librados: setting wanted keys 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 librados: calling monclient init 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient: init 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient(hunting): picked mon.noname-c con 0x7fa2900a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient(hunting): picked mon.noname-b con 0x7fa29013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient(hunting): picked mon.noname-a con 0x7fa2901430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29013be70 con 0x7fa2901430c0 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa290131660 con 0x7fa29013f710 2024-03-25T18:10:43.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2901317f0 con 0x7fa2900a98f0 2024-03-25T18:10:43.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:43.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.279+0000 7fa297472640 10 monclient(hunting): authenticate will time out at 7352.542426s 2024-03-25T18:10:43.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.293+0000 7fa29520e640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fa2900a98f0 0x7fa290130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:57290/0 (socket says 172.21.15.46:57290) 2024-03-25T18:10:43.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.293+0000 7fa29520e640 1 -- 172.21.15.46:0/3457807481 learned_addr learned my addr 172.21.15.46:0/3457807481 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:43.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.294+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa2900a98f0 2024-03-25T18:10:43.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.294+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.303+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa2901430c0 2024-03-25T18:10:43.304 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.303+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.306 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.304+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa29013f710 2024-03-25T18:10:43.306 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.304+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255619173 0 0) 0x7fa2901317f0 con 0x7fa2900a98f0 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:10:43.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.316+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274004eb0 con 0x7fa2900a98f0 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363513418 0 0) 0x7fa29013be70 con 0x7fa2901430c0 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:10:43.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.317+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2901317f0 con 0x7fa2901430c0 2024-03-25T18:10:43.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674675820 0 0) 0x7fa290131660 con 0x7fa29013f710 2024-03-25T18:10:43.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:10:43.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.326+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29013be70 con 0x7fa29013f710 2024-03-25T18:10:43.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.331+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3246922910 0 0) 0x7fa274004eb0 con 0x7fa2900a98f0 2024-03-25T18:10:43.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa290131660 con 0x7fa2900a98f0 2024-03-25T18:10:43.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa28c001bc0 con 0x7fa2900a98f0 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:43.334 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.335 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.332+0000 7fa285ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:43.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218261422 0 0) 0x7fa2901317f0 con 0x7fa2901430c0 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa274004eb0 con 0x7fa2901430c0 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa288001b50 con 0x7fa2901430c0 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:43.340 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1740721323 0 0) 0x7fa290131660 con 0x7fa2900a98f0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 >> v1:172.21.15.164:6789/0 conn(0x7fa29013f710 legacy=0x7fa2901d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 >> v1:172.21.15.46:6789/0 conn(0x7fa2901430c0 legacy=0x7fa2901e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 monclient: found mon.c 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290131980 con 0x7fa2900a98f0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 5 monclient: authenticate success, global_id 4316 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 10 monclient: _renew_subs 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa290135c60 con 0x7fa2900a98f0 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 10 monclient: _renew_subs 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2900b7770 con 0x7fa2900a98f0 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.337+0000 7fa297472640 1 librados: init done 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.338+0000 7fa297472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.338+0000 7fa297472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.338+0000 7fa297472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.338+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2901371d0 con 0x7fa2900a98f0 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa28c0011f0 con 0x7fa2900a98f0 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa28c004cb0 con 0x7fa2900a98f0 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa285ffb640 10 monclient: dump: 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:43.681 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.679+0000 7fa2877fe640 4 set_mon_vals no callback set 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.680+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fa28c01ad00 con 0x7fa2900a98f0 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.681+0000 7fa294a0d640 10 monclient: get_auth_request con 0x7fa27406df60 auth_method 0 2024-03-25T18:10:43.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.681+0000 7fa285ffb640 10 client.4316.objecter ms_handle_connect 0x7fa27406df60 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fa28c0549f0 con 0x7fa2900a98f0 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 10 client.4316.objecter ms_dispatch 0x7fa290000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa28c050330 con 0x7fa2900a98f0 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.687+0000 7fa285ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:43.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.921+0000 7fa297472640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.921+0000 7fa297472640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.921+0000 7fa297472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:10:43.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.921+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa290074cc0 con 0x7fa2900a98f0 2024-03-25T18:10:43.947 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.945+0000 7fa285ffb640 1 -- 172.21.15.46:0/3457807481 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fa28c052430 con 0x7fa2900a98f0 2024-03-25T18:10:43.947 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.945+0000 7fa285ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.947 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.945+0000 7fa285ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:43.947 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:43.947 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:43.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 10 librados: watch_flush enter 2024-03-25T18:10:43.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 10 librados: watch_flush exit 2024-03-25T18:10:43.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 10 client.4316.objecter successfully canceled tick 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 >> v1:172.21.15.164:6800/1612084612 conn(0x7fa27406df60 legacy=0x7fa274070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 10 monclient: shutdown 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 >> v1:172.21.15.46:6790/0 conn(0x7fa2900a98f0 legacy=0x7fa290130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 shutdown_connections 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.947+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 >> 172.21.15.46:0/3457807481 conn(0x7fa2900b7ea0 msgr2=0x7fa290074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.948+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 shutdown_connections 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.948+0000 7fa297472640 1 -- 172.21.15.46:0/3457807481 wait complete. 2024-03-25T18:10:43.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:43.948+0000 7fa297472640 1 librados: shutdown 2024-03-25T18:10:43.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:50.766 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:50.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.883+0000 7f7143d2b640 10 monclient: get_monmap_and_config 2024-03-25T18:10:50.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.883+0000 7f7143d2b640 10 monclient: build_initial_monmap 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.884+0000 7f7143d2b640 10 monclient: monmap: 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:50.885658+0000 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:50.885658+0000 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:50.886 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:50.887 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:50.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.884+0000 7f7143d2b640 1 Processor -- start 2024-03-25T18:10:50.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.884+0000 7f7143d2b640 1 -- start start 2024-03-25T18:10:50.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.884+0000 7f7143d2b640 10 monclient: init 2024-03-25T18:10:50.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f713c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f713c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f713c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.885+0000 7f7143d2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c142520 con 0x7f713c1430c0 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.886+0000 7f7143d2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0b28a0 con 0x7f713c0a98f0 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.886+0000 7f7143d2b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0b2a30 con 0x7f713c13f710 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.886+0000 7f7143d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:50.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.886+0000 7f7143d2b640 10 monclient(hunting): authenticate will time out at 7360.148615s 2024-03-25T18:10:50.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.899+0000 7f7141ac7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f713c0a98f0 0x7f713c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:49796/0 (socket says 172.21.15.46:49796) 2024-03-25T18:10:50.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.899+0000 7f7141ac7640 1 -- 172.21.15.46:0/4286799015 learned_addr learned my addr 172.21.15.46:0/4286799015 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:50.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426689219 0 0) 0x7f713c0b28a0 con 0x7f713c0a98f0 2024-03-25T18:10:50.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:10:50.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.921+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7128003520 con 0x7f713c0a98f0 2024-03-25T18:10:50.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.926+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412054657 0 0) 0x7f713c0b2a30 con 0x7f713c13f710 2024-03-25T18:10:50.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.927+0000 7f7140ac5640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.927+0000 7f7140ac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.927+0000 7f7140ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.927+0000 7f7140ac5640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:10:50.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.927+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c0b28a0 con 0x7f713c13f710 2024-03-25T18:10:50.932 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974754045 0 0) 0x7f713c142520 con 0x7f713c1430c0 2024-03-25T18:10:50.932 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.932 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:10:50.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.930+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c0b2a30 con 0x7f713c1430c0 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1325723361 0 0) 0x7f7128003520 con 0x7f713c0a98f0 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 >> v1:172.21.15.46:6790/0 conn(0x7f713c13f710 legacy=0x7f713c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 >> v1:172.21.15.46:6789/0 conn(0x7f713c1430c0 legacy=0x7f713c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 10 monclient: _finish_hunting 0 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 monclient: found mon.noname-c 2024-03-25T18:10:50.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0b44c0 con 0x7f713c0a98f0 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 10 monclient: _finish_auth 0 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 10 monclient: _check_auth_tickets 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7138001690 con 0x7f713c0a98f0 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.939+0000 7f7140ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.940+0000 7f7140ac5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.940+0000 7f7140ac5640 10 monclient: dump: 2024-03-25T18:10:50.943 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:50.944 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.940+0000 7f7143d2b640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:10:50.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.940+0000 7f7143d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:50.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71380017f0 con 0x7f713c0a98f0 2024-03-25T18:10:50.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:50.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 1 -- 172.21.15.46:0/4286799015 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7138001c00 con 0x7f713c0a98f0 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 4 set_mon_vals no callback set 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7140ac5640 10 monclient: dump: 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.957 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 10 monclient: shutdown 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 1 -- 172.21.15.46:0/4286799015 >> v1:172.21.15.164:6789/0 conn(0x7f713c0a98f0 legacy=0x7f713c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 1 -- 172.21.15.46:0/4286799015 shutdown_connections 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 1 -- 172.21.15.46:0/4286799015 >> 172.21.15.46:0/4286799015 conn(0x7f713c0b7ea0 msgr2=0x7f713c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:50.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 1 -- 172.21.15.46:0/4286799015 shutdown_connections 2024-03-25T18:10:50.959 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.954+0000 7f7143d2b640 1 -- 172.21.15.46:0/4286799015 wait complete. 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.955+0000 7f7143d2b640 10 monclient: build_initial_monmap 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.955+0000 7f7143d2b640 10 monclient: monmap: 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:50.961 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.955+0000 7f7143d2b640 1 librados: starting msgr at 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.955+0000 7f7143d2b640 1 librados: starting objecter 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.955+0000 7f7143d2b640 1 Processor -- start 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 -- start start 2024-03-25T18:10:50.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 librados: setting wanted keys 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 librados: calling monclient init 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient: init 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f713c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f713c13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f713c1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c139870 con 0x7f713c13f710 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c1d51f0 con 0x7f713c1430c0 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c138830 con 0x7f713c0a98f0 2024-03-25T18:10:50.963 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:50.964 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.956+0000 7f7143d2b640 10 monclient(hunting): authenticate will time out at 7360.219259s 2024-03-25T18:10:50.964 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.963+0000 7f71422c8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f713c1430c0 0x7f713c1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:49846/0 (socket says 172.21.15.46:49846) 2024-03-25T18:10:50.964 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.963+0000 7f71422c8640 1 -- 172.21.15.46:0/1246284082 learned_addr learned my addr 172.21.15.46:0/1246284082 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:50.966 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.964+0000 7f71327fc640 10 client.?.objecter ms_handle_connect 0x7f713c1430c0 2024-03-25T18:10:50.966 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.964+0000 7f71327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.968+0000 7f71327fc640 10 client.?.objecter ms_handle_connect 0x7f713c13f710 2024-03-25T18:10:50.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.968+0000 7f71327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612832838 0 0) 0x7f713c1d51f0 con 0x7f713c1430c0 2024-03-25T18:10:50.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.973 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:10:50.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.971+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120004eb0 con 0x7f713c1430c0 2024-03-25T18:10:50.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345356268 0 0) 0x7f713c139870 con 0x7f713c13f710 2024-03-25T18:10:50.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:10:50.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.975+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c1d51f0 con 0x7f713c13f710 2024-03-25T18:10:50.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.976+0000 7f71327fc640 10 client.?.objecter ms_handle_connect 0x7f713c0a98f0 2024-03-25T18:10:50.978 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.976+0000 7f71327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 758551566 0 0) 0x7f7120004eb0 con 0x7f713c1430c0 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f713c139870 con 0x7f713c1430c0 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7134001c70 con 0x7f713c1430c0 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 10 monclient(hunting): dump: 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:50.985 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:50.986 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.983+0000 7f71327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:50.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2506496571 0 0) 0x7f713c139870 con 0x7f713c1430c0 2024-03-25T18:10:50.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 >> v1:172.21.15.46:6790/0 conn(0x7f713c0a98f0 legacy=0x7f713c138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 >> v1:172.21.15.46:6789/0 conn(0x7f713c13f710 legacy=0x7f713c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 1 monclient: found mon.b 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c1d77a0 con 0x7f713c1430c0 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f71327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:10:50.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 10 monclient: _renew_subs 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c1d83d0 con 0x7f713c1430c0 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 10 monclient: _renew_subs 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c138c80 con 0x7f713c1430c0 2024-03-25T18:10:50.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.989+0000 7f7143d2b640 1 librados: init done 2024-03-25T18:10:50.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.990+0000 7f7143d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:50.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.990+0000 7f7143d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:50.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.990+0000 7f7143d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:50.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:50.990+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c0627b0 con 0x7f713c1430c0 2024-03-25T18:10:51.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.260+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7134003970 con 0x7f713c1430c0 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7134003d00 con 0x7f713c1430c0 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 10 monclient: dump: 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:51.263 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:51.264 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:51.264 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:51.264 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:51.264 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:51.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f7133fff640 4 set_mon_vals no callback set 2024-03-25T18:10:51.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.261+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7134003f20 con 0x7f713c1430c0 2024-03-25T18:10:51.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.262+0000 7f7141ac7640 10 monclient: get_auth_request con 0x7f712006dd10 auth_method 0 2024-03-25T18:10:51.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.262+0000 7f71327fc640 10 client.4318.objecter ms_handle_connect 0x7f712006dd10 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7134054ae0 con 0x7f713c1430c0 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 10 client.4318.objecter ms_dispatch 0x7f713c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71340505f0 con 0x7f713c1430c0 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:51.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.267+0000 7f71327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:51.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.486+0000 7f7143d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.486+0000 7f7143d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.486+0000 7f7143d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:51.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.486+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f713c1d7930 con 0x7f713c1430c0 2024-03-25T18:10:51.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.511+0000 7f71327fc640 1 -- 172.21.15.46:0/1246284082 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7134052520 con 0x7f713c1430c0 2024-03-25T18:10:51.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.511+0000 7f71327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.511+0000 7f71327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:51.513 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:51.513 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.513+0000 7f7143d2b640 10 librados: watch_flush enter 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.513+0000 7f7143d2b640 10 librados: watch_flush exit 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.513+0000 7f7143d2b640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.513+0000 7f7143d2b640 10 client.4318.objecter successfully canceled tick 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 >> v1:172.21.15.164:6800/1612084612 conn(0x7f712006dd10 legacy=0x7f71200701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 10 monclient: shutdown 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 >> v1:172.21.15.164:6789/0 conn(0x7f713c1430c0 legacy=0x7f713c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 shutdown_connections 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 >> 172.21.15.46:0/1246284082 conn(0x7f713c0b7ea0 msgr2=0x7f713c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 shutdown_connections 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 -- 172.21.15.46:0/1246284082 wait complete. 2024-03-25T18:10:51.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:51.514+0000 7f7143d2b640 1 librados: shutdown 2024-03-25T18:10:51.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:58.334 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.450+0000 7fbcae3af640 10 monclient: get_monmap_and_config 2024-03-25T18:10:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.450+0000 7fbcae3af640 10 monclient: build_initial_monmap 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.450+0000 7fbcae3af640 10 monclient: monmap: 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:10:58.450879+0000 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:10:58.450879+0000 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:10:58.451 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.450+0000 7fbcae3af640 1 Processor -- start 2024-03-25T18:10:58.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 1 -- start start 2024-03-25T18:10:58.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient: init 2024-03-25T18:10:58.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-b con 0x7fbca80a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-c con 0x7fbca813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-a con 0x7fbca81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca8142520 con 0x7fbca81430c0 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80b28a0 con 0x7fbca813f710 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80b2a30 con 0x7fbca80a98f0 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:58.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.451+0000 7fbcae3af640 10 monclient(hunting): authenticate will time out at 7367.713289s 2024-03-25T18:10:58.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.457+0000 7fbca7fff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fbca80a98f0 0x7fbca80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:53238/0 (socket says 172.21.15.46:53238) 2024-03-25T18:10:58.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.457+0000 7fbca7fff640 1 -- 172.21.15.46:0/3565115137 learned_addr learned my addr 172.21.15.46:0/3565115137 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492021561 0 0) 0x7fbca80b2a30 con 0x7fbca80a98f0 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:10:58.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.472+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc90003520 con 0x7fbca80a98f0 2024-03-25T18:10:58.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710145043 0 0) 0x7fbca8142520 con 0x7fbca81430c0 2024-03-25T18:10:58.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:10:58.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.485+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca80b2a30 con 0x7fbca81430c0 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414286563 0 0) 0x7fbca80b28a0 con 0x7fbca813f710 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:10:58.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.489+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca8142520 con 0x7fbca813f710 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146074604 0 0) 0x7fbc90003520 con 0x7fbca80a98f0 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 >> v1:172.21.15.164:6789/0 conn(0x7fbca813f710 legacy=0x7fbca81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 >> v1:172.21.15.46:6789/0 conn(0x7fbca81430c0 legacy=0x7fbca814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:10:58.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80b44c0 con 0x7fbca80a98f0 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fbc98001690 con 0x7fbca80a98f0 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbca6ffd640 10 monclient: dump: 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:58.493 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:58.494 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:58.494 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:58.494 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbcae3af640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:10:58.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.491+0000 7fbcae3af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.505+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc980017f0 con 0x7fbca80a98f0 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.505+0000 7fbca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.505+0000 7fbca6ffd640 1 -- 172.21.15.46:0/3565115137 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fbc98001c00 con 0x7fbca80a98f0 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.505+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbca6ffd640 10 monclient: dump: 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:58.506 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbcae3af640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:58.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbcae3af640 4 set_mon_vals no callback set 2024-03-25T18:10:58.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbcae3af640 10 monclient: shutdown 2024-03-25T18:10:58.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbcae3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:58.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.506+0000 7fbcae3af640 1 -- 172.21.15.46:0/3565115137 >> v1:172.21.15.46:6790/0 conn(0x7fbca80a98f0 legacy=0x7fbca80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.507+0000 7fbcae3af640 1 -- 172.21.15.46:0/3565115137 shutdown_connections 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.507+0000 7fbcae3af640 1 -- 172.21.15.46:0/3565115137 >> 172.21.15.46:0/3565115137 conn(0x7fbca80b7ea0 msgr2=0x7fbca80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.507+0000 7fbcae3af640 1 -- 172.21.15.46:0/3565115137 shutdown_connections 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.507+0000 7fbcae3af640 1 -- 172.21.15.46:0/3565115137 wait complete. 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 10 monclient: build_initial_monmap 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 10 monclient: monmap: 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 librados: starting msgr at 2024-03-25T18:10:58.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 librados: starting objecter 2024-03-25T18:10:58.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 Processor -- start 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 -- start start 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 librados: setting wanted keys 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 1 librados: calling monclient init 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.508+0000 7fbcae3af640 10 monclient: init 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-c con 0x7fbca80a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-b con 0x7fbca813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient(hunting): picked mon.noname-a con 0x7fbca81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca806e9d0 con 0x7fbca81430c0 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca806ec40 con 0x7fbca813f710 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca806ef20 con 0x7fbca80a98f0 2024-03-25T18:10:58.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:58.512 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.509+0000 7fbcae3af640 10 monclient(hunting): authenticate will time out at 7367.771287s 2024-03-25T18:10:58.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.515+0000 7fbca77fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbca813f710 0x7fbca806f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:38240/0 (socket says 172.21.15.46:38240) 2024-03-25T18:10:58.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.515+0000 7fbca77fe640 1 -- 172.21.15.46:0/3100142796 learned_addr learned my addr 172.21.15.46:0/3100142796 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:10:58.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.516+0000 7fbca4ff9640 10 client.?.objecter ms_handle_connect 0x7fbca813f710 2024-03-25T18:10:58.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.516+0000 7fbca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.528+0000 7fbca4ff9640 10 client.?.objecter ms_handle_connect 0x7fbca80a98f0 2024-03-25T18:10:58.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.528+0000 7fbca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939033584 0 0) 0x7fbca806ec40 con 0x7fbca813f710 2024-03-25T18:10:58.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:10:58.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc94004eb0 con 0x7fbca813f710 2024-03-25T18:10:58.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 client.?.objecter ms_handle_connect 0x7fbca81430c0 2024-03-25T18:10:58.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.540+0000 7fbca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297271040 0 0) 0x7fbca806ef20 con 0x7fbca80a98f0 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:10:58.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.546+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca806ec40 con 0x7fbca80a98f0 2024-03-25T18:10:58.551 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4218318169 0 0) 0x7fbc94004eb0 con 0x7fbca813f710 2024-03-25T18:10:58.551 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca806ef20 con 0x7fbca813f710 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fbc9c001a10 con 0x7fbca813f710 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.551+0000 7fbca4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:58.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2930276073 0 0) 0x7fbca806ef20 con 0x7fbca813f710 2024-03-25T18:10:58.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 >> v1:172.21.15.46:6790/0 conn(0x7fbca80a98f0 legacy=0x7fbca813b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 >> v1:172.21.15.46:6789/0 conn(0x7fbca81430c0 legacy=0x7fbca81e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:58.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 1 monclient: found mon.b 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca806fda0 con 0x7fbca813f710 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 10 monclient: _renew_subs 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca814dad0 con 0x7fbca813f710 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 10 monclient: _renew_subs 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca806e110 con 0x7fbca813f710 2024-03-25T18:10:58.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.560+0000 7fbcae3af640 1 librados: init done 2024-03-25T18:10:58.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.562+0000 7fbcae3af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.562+0000 7fbcae3af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.562+0000 7fbcae3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:58.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.562+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca80627b0 con 0x7fbca813f710 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc9c001c70 con 0x7fbca813f710 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fbc9c0038c0 con 0x7fbca813f710 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca4ff9640 10 monclient: dump: 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:10:58.834 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.833+0000 7fbca67fc640 4 set_mon_vals no callback set 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.834+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fbc9c003ae0 con 0x7fbca813f710 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.834+0000 7fbca7fff640 10 monclient: get_auth_request con 0x7fbc9406dd10 auth_method 0 2024-03-25T18:10:58.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.835+0000 7fbca4ff9640 10 client.4324.objecter ms_handle_connect 0x7fbc9406dd10 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.838+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fbc9c0546c0 con 0x7fbca813f710 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.838+0000 7fbca4ff9640 10 client.4324.objecter ms_dispatch 0x7fbca8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.838+0000 7fbca4ff9640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.838+0000 7fbca4ff9640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.838+0000 7fbca4ff9640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:10:58.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.839+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc9c0501d0 con 0x7fbca813f710 2024-03-25T18:10:58.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.839+0000 7fbca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:58.839+0000 7fbca4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:59.011 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:34.928 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.023 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:34.940 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.056+0000 7fbcae3af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:59.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.056+0000 7fbcae3af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:59.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.056+0000 7fbcae3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:10:59.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.056+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca80b3e80 con 0x7fbca813f710 2024-03-25T18:10:59.058 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:34.976 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.078 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:34.949 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.091 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:35.008 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.094 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:34.965 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.107 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:35.025 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.117 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:34.988 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:59.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.126+0000 7fbca4ff9640 1 -- 172.21.15.46:0/3100142796 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fbc9c052100 con 0x7fbca813f710 2024-03-25T18:10:59.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.126+0000 7fbca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:59.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.126+0000 7fbca4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:59.127 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:10:59.127 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 10 librados: watch_flush enter 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 10 librados: watch_flush exit 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 >> v1:172.21.15.164:6800/1612084612 conn(0x7fbc9406dd10 legacy=0x7fbc940701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:59.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 10 monclient: shutdown 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 >> v1:172.21.15.164:6789/0 conn(0x7fbca813f710 legacy=0x7fbca806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 shutdown_connections 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 >> 172.21.15.46:0/3100142796 conn(0x7fbca80b7ea0 msgr2=0x7fbca8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 shutdown_connections 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 -- 172.21.15.46:0/3100142796 wait complete. 2024-03-25T18:10:59.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:10:59.128+0000 7fbcae3af640 1 librados: shutdown 2024-03-25T18:10:59.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:59.153 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:35.071 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:05.946 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.061+0000 7f41a5074640 10 monclient: get_monmap_and_config 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.061+0000 7f41a5074640 10 monclient: build_initial_monmap 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.062+0000 7f41a5074640 10 monclient: monmap: 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:06.062685+0000 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:06.062685+0000 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:06.063 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.062+0000 7f41a5074640 1 Processor -- start 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.062+0000 7f41a5074640 1 -- start start 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.062+0000 7f41a5074640 10 monclient: init 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-a con 0x7f41a00a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-b con 0x7f41a013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-c con 0x7f41a01430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a0142520 con 0x7f41a00a98f0 2024-03-25T18:11:06.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00b28a0 con 0x7f41a01430c0 2024-03-25T18:11:06.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00b2a30 con 0x7f41a013f710 2024-03-25T18:11:06.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:06.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.063+0000 7f41a5074640 10 monclient(hunting): authenticate will time out at 7375.325315s 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.076+0000 7f419ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f41a00a98f0 0x7f41a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51980/0 (socket says 172.21.15.46:51980) 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.076+0000 7f419ed9d640 1 -- 172.21.15.46:0/3761833497 learned_addr learned my addr 172.21.15.46:0/3761833497 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:06.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.083+0000 7f419e59c640 1 -- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6790/0 conn(0x7f41a013f710 legacy=0x7f41a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:11:06.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.083+0000 7f419e59c640 1 -- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6790/0 conn(0x7f41a013f710 legacy=0x7f41a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:11:06.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.083+0000 7f419e59c640 1 --1- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6790/0 conn(0x7f41a013f710 0x7f41a01429b0 :-1 s=OPENED pgs=13438 cs=1 l=1).handle_message read tag failed 2024-03-25T18:11:06.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.083+0000 7f419e59c640 1 --1- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6790/0 conn(0x7f41a013f710 0x7f41a01429b0 :-1 s=OPENED pgs=13438 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:11:06.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.083+0000 7f419dd9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.46:6790/0 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908743392 0 0) 0x7f41a00b28a0 con 0x7f41a01430c0 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:11:06.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.085+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4188003520 con 0x7f41a01430c0 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345348620 0 0) 0x7f41a0142520 con 0x7f41a00a98f0 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:11:06.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.090+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a00b28a0 con 0x7f41a00a98f0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013155618 0 0) 0x7f4188003520 con 0x7f41a01430c0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6790/0 conn(0x7f41a013f710 legacy=0x7f41a01429b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 >> v1:172.21.15.46:6789/0 conn(0x7f41a00a98f0 legacy=0x7f41a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00b44c0 con 0x7f41a01430c0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4198001a10 con 0x7f41a01430c0 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:06.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f419dd9b640 10 monclient: dump: 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f41a5074640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:11:06.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.107+0000 7f41a5074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4198001b70 con 0x7f41a01430c0 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 1 -- 172.21.15.46:0/3761833497 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4198001f80 con 0x7f41a01430c0 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f41a5074640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f41a5074640 4 set_mon_vals no callback set 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f419dd9b640 10 monclient: dump: 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:06.125 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f41a5074640 10 monclient: shutdown 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f41a5074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.124+0000 7f41a5074640 1 -- 172.21.15.46:0/3761833497 >> v1:172.21.15.164:6789/0 conn(0x7f41a01430c0 legacy=0x7f41a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.125+0000 7f41a5074640 1 -- 172.21.15.46:0/3761833497 shutdown_connections 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.125+0000 7f41a5074640 1 -- 172.21.15.46:0/3761833497 >> 172.21.15.46:0/3761833497 conn(0x7f41a00b7ea0 msgr2=0x7f41a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:06.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.125+0000 7f41a5074640 1 -- 172.21.15.46:0/3761833497 shutdown_connections 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.125+0000 7f41a5074640 1 -- 172.21.15.46:0/3761833497 wait complete. 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.126+0000 7f41a5074640 10 monclient: build_initial_monmap 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.126+0000 7f41a5074640 10 monclient: monmap: 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:06.127 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.126+0000 7f41a5074640 1 librados: starting msgr at 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.126+0000 7f41a5074640 1 librados: starting objecter 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.126+0000 7f41a5074640 1 Processor -- start 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 1 -- start start 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 1 librados: setting wanted keys 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 1 librados: calling monclient init 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 10 monclient: init 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.127+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-b con 0x7f41a00a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-a con 0x7f41a013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 10 monclient(hunting): picked mon.noname-c con 0x7f41a01430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a0138ef0 con 0x7f41a013f710 2024-03-25T18:11:06.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a0136bf0 con 0x7f41a00a98f0 2024-03-25T18:11:06.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a0137d00 con 0x7f41a01430c0 2024-03-25T18:11:06.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:06.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.128+0000 7f41a5074640 10 monclient(hunting): authenticate will time out at 7375.389841s 2024-03-25T18:11:06.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.138+0000 7f419ed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f41a00a98f0 0x7f41a01375f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47046/0 (socket says 172.21.15.46:47046) 2024-03-25T18:11:06.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.138+0000 7f419ed9d640 1 -- 172.21.15.46:0/548717103 learned_addr learned my addr 172.21.15.46:0/548717103 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:06.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.139+0000 7f41877fe640 10 client.?.objecter ms_handle_connect 0x7f41a00a98f0 2024-03-25T18:11:06.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.139+0000 7f41877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:06.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.141+0000 7f41877fe640 10 client.?.objecter ms_handle_connect 0x7f41a013f710 2024-03-25T18:11:06.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.141+0000 7f41877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:06.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.145+0000 7f41877fe640 10 client.?.objecter ms_handle_connect 0x7f41a01430c0 2024-03-25T18:11:06.146 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.145+0000 7f41877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966629240 0 0) 0x7f41a0138ef0 con 0x7f41a013f710 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:11:06.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.147+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c004eb0 con 0x7f41a013f710 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.148+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288208202 0 0) 0x7f41a0136bf0 con 0x7f41a00a98f0 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.148+0000 7f41877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.148+0000 7f41877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.148+0000 7f41877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.148+0000 7f41877fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:11:06.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.149+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0138ef0 con 0x7f41a00a98f0 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200581151 0 0) 0x7f41a0137d00 con 0x7f41a01430c0 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:11:06.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.151+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0136bf0 con 0x7f41a01430c0 2024-03-25T18:11:06.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764461535 0 0) 0x7f41a0138ef0 con 0x7f41a00a98f0 2024-03-25T18:11:06.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41a0137d00 con 0x7f41a00a98f0 2024-03-25T18:11:06.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f41900019b0 con 0x7f41a00a98f0 2024-03-25T18:11:06.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 10 monclient(hunting): dump: 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.155+0000 7f41877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:06.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3176354669 0 0) 0x7f41a0136bf0 con 0x7f41a01430c0 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41a0138ef0 con 0x7f41a01430c0 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4198001e80 con 0x7f41a01430c0 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.162+0000 7f41877fe640 10 monclient(hunting): dump: 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:06.163 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148783305 0 0) 0x7f41a0137d00 con 0x7f41a00a98f0 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 >> v1:172.21.15.46:6790/0 conn(0x7f41a01430c0 legacy=0x7f41a01e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 >> v1:172.21.15.46:6789/0 conn(0x7f41a013f710 legacy=0x7f41a0133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 1 monclient: found mon.b 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0133810 con 0x7f41a00a98f0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 10 monclient: _renew_subs 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a0133a60 con 0x7f41a00a98f0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 10 monclient: _renew_subs 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a01372f0 con 0x7f41a00a98f0 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.163+0000 7f41a5074640 1 librados: init done 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.164+0000 7f41a5074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.164+0000 7f41a5074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.164+0000 7f41a5074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.164+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a00627b0 con 0x7f41a00a98f0 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41900047e0 con 0x7f41a00a98f0 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4190004af0 con 0x7f41a00a98f0 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f41877fe640 10 monclient: dump: 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:06.450 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.449+0000 7f419d59a640 4 set_mon_vals no callback set 2024-03-25T18:11:06.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.450+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f419001ab00 con 0x7f41a00a98f0 2024-03-25T18:11:06.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.450+0000 7f419e59c640 10 monclient: get_auth_request con 0x7f418c06df80 auth_method 0 2024-03-25T18:11:06.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.451+0000 7f41877fe640 10 client.4330.objecter ms_handle_connect 0x7f418c06df80 2024-03-25T18:11:06.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.456+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f41900547c0 con 0x7f41a00a98f0 2024-03-25T18:11:06.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.456+0000 7f41877fe640 10 client.4330.objecter ms_dispatch 0x7f41a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:06.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.456+0000 7f41877fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:06.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.456+0000 7f41877fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:06.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.456+0000 7f41877fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:11:06.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.457+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4190050100 con 0x7f41a00a98f0 2024-03-25T18:11:06.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.458+0000 7f41877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.458+0000 7f41877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:06.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.691+0000 7f41a5074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.691+0000 7f41a5074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.691+0000 7f41a5074640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:06.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.691+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41a00008d0 con 0x7f41a00a98f0 2024-03-25T18:11:06.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.712+0000 7f41877fe640 1 -- 172.21.15.46:0/548717103 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f4190052200 con 0x7f41a00a98f0 2024-03-25T18:11:06.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.712+0000 7f41877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.712+0000 7f41877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:06.713 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:06.713 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 10 librados: watch_flush enter 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 10 librados: watch_flush exit 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 >> v1:172.21.15.164:6800/1612084612 conn(0x7f418c06df80 legacy=0x7f418c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 10 monclient: shutdown 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 >> v1:172.21.15.164:6789/0 conn(0x7f41a00a98f0 legacy=0x7f41a01375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 shutdown_connections 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 >> 172.21.15.46:0/548717103 conn(0x7f41a00b7ea0 msgr2=0x7f41a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 shutdown_connections 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 -- 172.21.15.46:0/548717103 wait complete. 2024-03-25T18:11:06.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:06.715+0000 7f41a5074640 1 librados: shutdown 2024-03-25T18:11:06.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:13.534 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.649+0000 7f5ab5a06640 10 monclient: get_monmap_and_config 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.649+0000 7f5ab5a06640 10 monclient: build_initial_monmap 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.649+0000 7f5ab5a06640 10 monclient: monmap: 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:13.650203+0000 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:13.650203+0000 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 1 Processor -- start 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 1 -- start start 2024-03-25T18:11:13.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient: init 2024-03-25T18:11:13.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:13.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:13.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab00a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab01430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab0142520 con 0x7f5ab00a98f0 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.650+0000 7f5ab5a06640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00b28a0 con 0x7f5ab013f710 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.651+0000 7f5ab5a06640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00b2a30 con 0x7f5ab01430c0 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.651+0000 7f5ab5a06640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:13.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.651+0000 7f5ab5a06640 10 monclient(hunting): authenticate will time out at 7382.912566s 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.655+0000 7f5aae7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5ab013f710 0x7f5ab01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45694/0 (socket says 172.21.15.46:45694) 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.655+0000 7f5aae7fc640 1 -- 172.21.15.46:0/854188972 learned_addr learned my addr 172.21.15.46:0/854188972 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386238255 0 0) 0x7f5ab00b28a0 con 0x7f5ab013f710 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:11:13.667 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.666+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98003520 con 0x7f5ab013f710 2024-03-25T18:11:13.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373069833 0 0) 0x7f5ab0142520 con 0x7f5ab00a98f0 2024-03-25T18:11:13.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:11:13.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.673+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab00b28a0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525602637 0 0) 0x7f5ab00b28a0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 >> v1:172.21.15.46:6790/0 conn(0x7f5ab01430c0 legacy=0x7f5ab014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 >> v1:172.21.15.164:6789/0 conn(0x7f5ab013f710 legacy=0x7f5ab01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 monclient: found mon.noname-a 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:13.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00b44c0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa0001670 con 0x7f5ab00a98f0 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5aadffb640 10 monclient: dump: 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.681 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5ab5a06640 5 monclient: authenticate success, global_id 4329 2024-03-25T18:11:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.679+0000 7f5ab5a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa00017d0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 1 -- 172.21.15.46:0/854188972 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa0001ba0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5aadffb640 10 monclient: dump: 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.689 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5ab5a06640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:13.690 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.688+0000 7f5ab5a06640 4 set_mon_vals no callback set 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 10 monclient: shutdown 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 1 -- 172.21.15.46:0/854188972 >> v1:172.21.15.46:6789/0 conn(0x7f5ab00a98f0 legacy=0x7f5ab00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 1 -- 172.21.15.46:0/854188972 shutdown_connections 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 1 -- 172.21.15.46:0/854188972 >> 172.21.15.46:0/854188972 conn(0x7f5ab00b7ea0 msgr2=0x7f5ab00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:13.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 1 -- 172.21.15.46:0/854188972 shutdown_connections 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.689+0000 7f5ab5a06640 1 -- 172.21.15.46:0/854188972 wait complete. 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 10 monclient: build_initial_monmap 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 10 monclient: monmap: 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 librados: starting msgr at 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 librados: starting objecter 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 Processor -- start 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 -- start start 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 librados: setting wanted keys 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 1 librados: calling monclient init 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.690+0000 7f5ab5a06640 10 monclient: init 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab00a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab01430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab01da8f0 con 0x7f5ab01430c0 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab01399f0 con 0x7f5ab013f710 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab01db770 con 0x7f5ab00a98f0 2024-03-25T18:11:13.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:13.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.691+0000 7f5ab5a06640 10 monclient(hunting): authenticate will time out at 7382.953252s 2024-03-25T18:11:13.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.705+0000 7f5aae7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5ab013f710 0x7f5ab01381f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45722/0 (socket says 172.21.15.46:45722) 2024-03-25T18:11:13.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.705+0000 7f5aae7fc640 1 -- 172.21.15.46:0/4063669508 learned_addr learned my addr 172.21.15.46:0/4063669508 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:13.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.706+0000 7f5a97fff640 10 client.?.objecter ms_handle_connect 0x7f5ab01430c0 2024-03-25T18:11:13.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.706+0000 7f5a97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.707+0000 7f5a97fff640 10 client.?.objecter ms_handle_connect 0x7f5ab013f710 2024-03-25T18:11:13.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.707+0000 7f5a97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.709+0000 7f5a97fff640 10 client.?.objecter ms_handle_connect 0x7f5ab00a98f0 2024-03-25T18:11:13.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.709+0000 7f5a97fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876283359 0 0) 0x7f5ab01da8f0 con 0x7f5ab01430c0 2024-03-25T18:11:13.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.715 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.715 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:11:13.715 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.714+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9c004eb0 con 0x7f5ab01430c0 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741420455 0 0) 0x7f5ab01399f0 con 0x7f5ab013f710 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:11:13.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.721+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab01da8f0 con 0x7f5ab013f710 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490368082 0 0) 0x7f5a9c004eb0 con 0x7f5ab01430c0 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab01399f0 con 0x7f5ab01430c0 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa8001b20 con 0x7f5ab01430c0 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 10 monclient(hunting): dump: 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:13.730 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.729+0000 7f5a97fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:13.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113112979 0 0) 0x7f5ab01db770 con 0x7f5ab00a98f0 2024-03-25T18:11:13.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:11:13.732 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.731+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9c004eb0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218889778 0 0) 0x7f5ab01da8f0 con 0x7f5ab013f710 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab01db770 con 0x7f5ab013f710 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa40018d0 con 0x7f5ab013f710 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.735+0000 7f5a97fff640 10 monclient(hunting): dump: 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.736 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:13.737 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:13.737 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:13.737 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248480691 0 0) 0x7f5a9c004eb0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab01da8f0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa0001ae0 con 0x7f5ab00a98f0 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.746+0000 7f5a97fff640 10 monclient(hunting): dump: 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:13.747 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.748 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:13.748 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:13.748 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:13.748 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:13.748 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572333455 0 0) 0x7f5ab01db770 con 0x7f5ab013f710 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 >> v1:172.21.15.46:6790/0 conn(0x7f5ab00a98f0 legacy=0x7f5ab01db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 >> v1:172.21.15.46:6789/0 conn(0x7f5ab01430c0 legacy=0x7f5ab01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 1 monclient: found mon.b 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab01dab40 con 0x7f5ab013f710 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5a97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:13.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 10 monclient: _renew_subs 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab0137870 con 0x7f5ab013f710 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 10 monclient: _renew_subs 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab0138ae0 con 0x7f5ab013f710 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.757+0000 7f5ab5a06640 1 librados: init done 2024-03-25T18:11:13.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.758+0000 7f5ab5a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:13.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.758+0000 7f5ab5a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:13.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.758+0000 7f5ab5a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:13.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:13.758+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab01dad80 con 0x7f5ab013f710 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa4001d00 con 0x7f5ab013f710 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5aa40047e0 con 0x7f5ab013f710 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5a97fff640 10 monclient: dump: 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:14.037 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:14.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.036+0000 7f5aad7fa640 4 set_mon_vals no callback set 2024-03-25T18:11:14.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.037+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f5aa4004a00 con 0x7f5ab013f710 2024-03-25T18:11:14.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.038+0000 7f5aaeffd640 10 monclient: get_auth_request con 0x7f5a9c06e0c0 auth_method 0 2024-03-25T18:11:14.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.038+0000 7f5a97fff640 10 client.4336.objecter ms_handle_connect 0x7f5a9c06e0c0 2024-03-25T18:11:14.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f5aa4054650 con 0x7f5ab013f710 2024-03-25T18:11:14.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 10 client.4336.objecter ms_dispatch 0x7f5ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5aa404ff90 con 0x7f5ab013f710 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:14.043 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.042+0000 7f5a97fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:14.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.273+0000 7f5ab5a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.273+0000 7f5ab5a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.273+0000 7f5ab5a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:14.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.273+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab00627b0 con 0x7f5ab013f710 2024-03-25T18:11:14.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.314+0000 7f5a97fff640 1 -- 172.21.15.46:0/4063669508 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f5aa4052090 con 0x7f5ab013f710 2024-03-25T18:11:14.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.314+0000 7f5a97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.314+0000 7f5a97fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:14.315 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:14.315 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:14.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 10 librados: watch_flush enter 2024-03-25T18:11:14.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 10 librados: watch_flush exit 2024-03-25T18:11:14.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:11:14.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 >> v1:172.21.15.164:6800/1612084612 conn(0x7f5a9c06e0c0 legacy=0x7f5a9c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 10 monclient: shutdown 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 >> v1:172.21.15.164:6789/0 conn(0x7f5ab013f710 legacy=0x7f5ab01381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 shutdown_connections 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 >> 172.21.15.46:0/4063669508 conn(0x7f5ab00b7ea0 msgr2=0x7f5ab013d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 shutdown_connections 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 -- 172.21.15.46:0/4063669508 wait complete. 2024-03-25T18:11:14.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:14.317+0000 7f5ab5a06640 1 librados: shutdown 2024-03-25T18:11:14.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:18.846 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:54.764 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.867 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:54.784 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.902 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:02:54.819 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.918 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:54.789 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.925 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:54.796 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.929 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:54.847 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.936 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:54.854 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.961 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:02:54.832 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:18.974 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:02:54.892 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:21.136 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:21.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.255+0000 7f4a7a1e9640 10 monclient: get_monmap_and_config 2024-03-25T18:11:21.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.255+0000 7f4a7a1e9640 10 monclient: build_initial_monmap 2024-03-25T18:11:21.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.255+0000 7f4a7a1e9640 10 monclient: monmap: 2024-03-25T18:11:21.256 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:21.256395+0000 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:21.256395+0000 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.256+0000 7f4a7a1e9640 1 Processor -- start 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.256+0000 7f4a7a1e9640 1 -- start start 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.256+0000 7f4a7a1e9640 10 monclient: init 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4a740a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4a741430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a74142520 con 0x7f4a741430c0 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a740b28a0 con 0x7f4a740a98f0 2024-03-25T18:11:21.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a740b2a30 con 0x7f4a7413f710 2024-03-25T18:11:21.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:21.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.257+0000 7f4a7a1e9640 10 monclient(hunting): authenticate will time out at 7390.519468s 2024-03-25T18:11:21.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.265+0000 7f4a73fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a741430c0 0x7f4a7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43770/0 (socket says 172.21.15.46:43770) 2024-03-25T18:11:21.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.265+0000 7f4a73fff640 1 -- 172.21.15.46:0/2713420838 learned_addr learned my addr 172.21.15.46:0/2713420838 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:21.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611259002 0 0) 0x7f4a74142520 con 0x7f4a741430c0 2024-03-25T18:11:21.276 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.277 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.277 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.277 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:11:21.277 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.276+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c003520 con 0x7f4a741430c0 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729431955 0 0) 0x7f4a740b28a0 con 0x7f4a740a98f0 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:11:21.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.284+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74142520 con 0x7f4a740a98f0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666887689 0 0) 0x7f4a5c003520 con 0x7f4a741430c0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 >> v1:172.21.15.46:6790/0 conn(0x7f4a7413f710 legacy=0x7f4a741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 >> v1:172.21.15.164:6789/0 conn(0x7f4a740a98f0 legacy=0x7f4a740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 monclient: found mon.noname-a 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a740b44c0 con 0x7f4a741430c0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4a6c001a10 con 0x7f4a741430c0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a727fc640 10 monclient: dump: 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a7a1e9640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.285+0000 7f4a7a1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:21.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a6c001b70 con 0x7f4a741430c0 2024-03-25T18:11:21.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 1 -- 172.21.15.46:0/2713420838 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4a6c001f80 con 0x7f4a741430c0 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a7a1e9640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a7a1e9640 4 set_mon_vals no callback set 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.290+0000 7f4a727fc640 10 monclient: dump: 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.291 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:21.292 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:21.292 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:21.292 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 10 monclient: shutdown 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2713420838 >> v1:172.21.15.46:6789/0 conn(0x7f4a741430c0 legacy=0x7f4a7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2713420838 shutdown_connections 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2713420838 >> 172.21.15.46:0/2713420838 conn(0x7f4a740b7ea0 msgr2=0x7f4a740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2713420838 shutdown_connections 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.291+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2713420838 wait complete. 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 10 monclient: build_initial_monmap 2024-03-25T18:11:21.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 10 monclient: monmap: 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 librados: starting msgr at 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 librados: starting objecter 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 Processor -- start 2024-03-25T18:11:21.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 -- start start 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 librados: setting wanted keys 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 1 librados: calling monclient init 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.292+0000 7f4a7a1e9640 10 monclient: init 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4a740a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4a7413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4a741430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a741d7df0 con 0x7f4a7413f710 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a741d7f80 con 0x7f4a741430c0 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7413b5c0 con 0x7f4a740a98f0 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:21.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.293+0000 7f4a7a1e9640 10 monclient(hunting): authenticate will time out at 7390.555268s 2024-03-25T18:11:21.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.302+0000 7f4a72ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a7413f710 0x7f4a74133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43802/0 (socket says 172.21.15.46:43802) 2024-03-25T18:11:21.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.302+0000 7f4a72ffd640 1 -- 172.21.15.46:0/2654393976 learned_addr learned my addr 172.21.15.46:0/2654393976 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:21.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.305+0000 7f4a5bfff640 10 client.?.objecter ms_handle_connect 0x7f4a7413f710 2024-03-25T18:11:21.306 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.305+0000 7f4a5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166645969 0 0) 0x7f4a741d7df0 con 0x7f4a7413f710 2024-03-25T18:11:21.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:11:21.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.309+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60004eb0 con 0x7f4a7413f710 2024-03-25T18:11:21.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.314+0000 7f4a5bfff640 10 client.?.objecter ms_handle_connect 0x7f4a741430c0 2024-03-25T18:11:21.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.314+0000 7f4a5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.318+0000 7f4a5bfff640 10 client.?.objecter ms_handle_connect 0x7f4a740a98f0 2024-03-25T18:11:21.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.318+0000 7f4a5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4257603588 0 0) 0x7f4a60004eb0 con 0x7f4a7413f710 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a741d7df0 con 0x7f4a7413f710 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4a68001a30 con 0x7f4a7413f710 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): dump: 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:21.324 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234217958 0 0) 0x7f4a741d7f80 con 0x7f4a741430c0 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:11:21.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.323+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60004eb0 con 0x7f4a741430c0 2024-03-25T18:11:21.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375971385 0 0) 0x7f4a60004eb0 con 0x7f4a741430c0 2024-03-25T18:11:21.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a741d7f80 con 0x7f4a741430c0 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4a6c001e80 con 0x7f4a741430c0 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.341+0000 7f4a5bfff640 10 monclient(hunting): dump: 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:21.342 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2436371567 0 0) 0x7f4a741d7df0 con 0x7f4a7413f710 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 >> v1:172.21.15.46:6790/0 conn(0x7f4a740a98f0 legacy=0x7f4a741d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 >> v1:172.21.15.164:6789/0 conn(0x7f4a741430c0 legacy=0x7f4a74135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 1 monclient: found mon.a 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:21.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7413b750 con 0x7f4a7413f710 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 10 monclient: _finish_auth 0 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 10 monclient: _renew_subs 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a74133710 con 0x7f4a7413f710 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 10 monclient: _renew_subs 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a741357a0 con 0x7f4a7413f710 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.342+0000 7f4a7a1e9640 1 librados: init done 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.343+0000 7f4a7a1e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.343+0000 7f4a7a1e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.343+0000 7f4a7a1e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.343+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a74138fd0 con 0x7f4a7413f710 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a68001cb0 con 0x7f4a7413f710 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4a680038c0 con 0x7f4a7413f710 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a5bfff640 10 monclient: dump: 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.654 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.653+0000 7f4a71ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:21.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.654+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f4a68003aa0 con 0x7f4a7413f710 2024-03-25T18:11:21.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.654+0000 7f4a737fe640 10 monclient: get_auth_request con 0x7f4a6006def0 auth_method 0 2024-03-25T18:11:21.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.655+0000 7f4a5bfff640 10 client.4338.objecter ms_handle_connect 0x7f4a6006def0 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f4a68054710 con 0x7f4a7413f710 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 10 client.4338.objecter ms_dispatch 0x7f4a74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a68050220 con 0x7f4a7413f710 2024-03-25T18:11:21.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.660+0000 7f4a5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:21.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.922+0000 7f4a7a1e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.922+0000 7f4a7a1e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.922+0000 7f4a7a1e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:21.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.922+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a74074cc0 con 0x7f4a7413f710 2024-03-25T18:11:21.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.949+0000 7f4a5bfff640 1 -- 172.21.15.46:0/2654393976 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f4a68052150 con 0x7f4a7413f710 2024-03-25T18:11:21.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.949+0000 7f4a5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.949+0000 7f4a5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:21.950 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:21.950 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 10 librados: watch_flush enter 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 10 librados: watch_flush exit 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 10 client.4338.objecter successfully canceled tick 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 >> v1:172.21.15.164:6800/1612084612 conn(0x7f4a6006def0 legacy=0x7f4a600703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 10 monclient: shutdown 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 >> v1:172.21.15.46:6789/0 conn(0x7f4a7413f710 legacy=0x7f4a74133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 shutdown_connections 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 >> 172.21.15.46:0/2654393976 conn(0x7f4a740b7ea0 msgr2=0x7f4a74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:21.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 shutdown_connections 2024-03-25T18:11:21.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 -- 172.21.15.46:0/2654393976 wait complete. 2024-03-25T18:11:21.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:21.952+0000 7f4a7a1e9640 1 librados: shutdown 2024-03-25T18:11:21.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:28.772 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.887+0000 7f4af419a640 10 monclient: get_monmap_and_config 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.887+0000 7f4af419a640 10 monclient: build_initial_monmap 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.887+0000 7f4af419a640 10 monclient: monmap: 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:28.888292+0000 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:28.888292+0000 2024-03-25T18:11:28.888 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:28.889 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:28.889 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:28.889 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:28.889 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:28.889 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:28.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.888+0000 7f4af419a640 1 Processor -- start 2024-03-25T18:11:28.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.888+0000 7f4af419a640 1 -- start start 2024-03-25T18:11:28.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.888+0000 7f4af419a640 10 monclient: init 2024-03-25T18:11:28.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-b con 0x7f4aec0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-a con 0x7f4aec13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-c con 0x7f4aec1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec142520 con 0x7f4aec13f710 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0b28a0 con 0x7f4aec1430c0 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0b2a30 con 0x7f4aec0a98f0 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:28.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.889+0000 7f4af419a640 10 monclient(hunting): authenticate will time out at 7398.151121s 2024-03-25T18:11:28.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.910+0000 7f4af1f36640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f4aec0a98f0 0x7f4aec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:51730/0 (socket says 172.21.15.46:51730) 2024-03-25T18:11:28.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.910+0000 7f4af1f36640 1 -- 172.21.15.46:0/4196290172 learned_addr learned my addr 172.21.15.46:0/4196290172 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:28.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877915477 0 0) 0x7f4aec0b28a0 con 0x7f4aec1430c0 2024-03-25T18:11:28.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:11:28.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.926+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad8003520 con 0x7f4aec1430c0 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880308347 0 0) 0x7f4aec0b2a30 con 0x7f4aec0a98f0 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec0b28a0 con 0x7f4aec0a98f0 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909611791 0 0) 0x7f4aec142520 con 0x7f4aec13f710 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.935 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:11:28.936 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.934+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec0b2a30 con 0x7f4aec13f710 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881597161 0 0) 0x7f4aec0b2a30 con 0x7f4aec13f710 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 >> v1:172.21.15.46:6790/0 conn(0x7f4aec0a98f0 legacy=0x7f4aec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 >> v1:172.21.15.164:6789/0 conn(0x7f4aec1430c0 legacy=0x7f4aec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: _finish_hunting 0 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 monclient: found mon.noname-a 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:11:28.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec0b44c0 con 0x7f4aec13f710 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: _finish_auth 0 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: _check_auth_tickets 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4adc0015b0 con 0x7f4aec13f710 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af0f34640 10 monclient: dump: 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af419a640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.944+0000 7f4af419a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:28.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4adc001710 con 0x7f4aec13f710 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 1 -- 172.21.15.46:0/4196290172 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4adc001b60 con 0x7f4aec13f710 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af419a640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af419a640 4 set_mon_vals no callback set 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.952+0000 7f4af0f34640 10 monclient: dump: 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.954 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:28.954 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:28.954 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:28.954 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:28.954 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 10 monclient: shutdown 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 1 -- 172.21.15.46:0/4196290172 >> v1:172.21.15.46:6789/0 conn(0x7f4aec13f710 legacy=0x7f4aec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 1 -- 172.21.15.46:0/4196290172 shutdown_connections 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 1 -- 172.21.15.46:0/4196290172 >> 172.21.15.46:0/4196290172 conn(0x7f4aec0b7ea0 msgr2=0x7f4aec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 1 -- 172.21.15.46:0/4196290172 shutdown_connections 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 1 -- 172.21.15.46:0/4196290172 wait complete. 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 10 monclient: build_initial_monmap 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.953+0000 7f4af419a640 10 monclient: monmap: 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:28.956 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 librados: starting msgr at 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 librados: starting objecter 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 Processor -- start 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 -- start start 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 librados: setting wanted keys 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 1 librados: calling monclient init 2024-03-25T18:11:28.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.954+0000 7f4af419a640 10 monclient: init 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-b con 0x7f4aec0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-c con 0x7f4aec13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient(hunting): picked mon.noname-a con 0x7f4aec1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec13a570 con 0x7f4aec1430c0 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec13a0e0 con 0x7f4aec0a98f0 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0b7c60 con 0x7f4aec13f710 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:28.958 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.955+0000 7f4af419a640 10 monclient(hunting): authenticate will time out at 7398.216923s 2024-03-25T18:11:28.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.974+0000 7f4af1f36640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4aec0a98f0 0x7f4aec0b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:35016/0 (socket says 172.21.15.46:35016) 2024-03-25T18:11:28.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.974+0000 7f4af1f36640 1 -- 172.21.15.46:0/116261575 learned_addr learned my addr 172.21.15.46:0/116261575 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:28.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.976+0000 7f4ae27fc640 10 client.?.objecter ms_handle_connect 0x7f4aec0a98f0 2024-03-25T18:11:28.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.976+0000 7f4ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768663309 0 0) 0x7f4aec13a0e0 con 0x7f4aec0a98f0 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0004eb0 con 0x7f4aec0a98f0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 client.?.objecter ms_handle_connect 0x7f4aec13f710 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.982+0000 7f4ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:28.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.987+0000 7f4ae27fc640 10 client.?.objecter ms_handle_connect 0x7f4aec1430c0 2024-03-25T18:11:28.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:28.987+0000 7f4ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:29.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747722450 0 0) 0x7f4aec13a570 con 0x7f4aec1430c0 2024-03-25T18:11:29.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:29.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:29.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:29.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:11:29.004 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.003+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec13a0e0 con 0x7f4aec1430c0 2024-03-25T18:11:29.006 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620321977 0 0) 0x7f4aec0b7c60 con 0x7f4aec13f710 2024-03-25T18:11:29.006 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:29.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:29.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:29.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:11:29.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.005+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec13a570 con 0x7f4aec13f710 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 390757243 0 0) 0x7f4ad0004eb0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aec0b7c60 con 0x7f4aec0a98f0 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4ae80019b0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 10 monclient(hunting): dump: 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.011 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:29.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.010+0000 7f4ae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:29.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2291586541 0 0) 0x7f4aec13a0e0 con 0x7f4aec1430c0 2024-03-25T18:11:29.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad0004eb0 con 0x7f4aec1430c0 2024-03-25T18:11:29.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4ae4001a70 con 0x7f4aec1430c0 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 10 monclient(hunting): dump: 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:29.022 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3076009476 0 0) 0x7f4aec0b7c60 con 0x7f4aec0a98f0 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 >> v1:172.21.15.46:6790/0 conn(0x7f4aec13f710 legacy=0x7f4aec12d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 >> v1:172.21.15.46:6789/0 conn(0x7f4aec1430c0 legacy=0x7f4aec1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 monclient: found mon.b 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.021+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec139d20 con 0x7f4aec0a98f0 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4ae27fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4ae27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:29.024 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4ae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 10 monclient: _renew_subs 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec12d9e0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 10 monclient: _renew_subs 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec12f650 con 0x7f4aec0a98f0 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.022+0000 7f4af419a640 1 librados: init done 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.023+0000 7f4af419a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.023+0000 7f4af419a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.023+0000 7f4af419a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:29.025 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.023+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec0b3ef0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae80047e0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f4ae8004af0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae27fc640 10 monclient: dump: 2024-03-25T18:11:29.314 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:29.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.313+0000 7f4ae3fff640 4 set_mon_vals no callback set 2024-03-25T18:11:29.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.314+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f4ae801ab00 con 0x7f4aec0a98f0 2024-03-25T18:11:29.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.315+0000 7f4af1735640 10 monclient: get_auth_request con 0x7f4ad006df70 auth_method 0 2024-03-25T18:11:29.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.315+0000 7f4ae27fc640 10 client.4348.objecter ms_handle_connect 0x7f4ad006df70 2024-03-25T18:11:29.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.316+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f4ae8054870 con 0x7f4aec0a98f0 2024-03-25T18:11:29.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.316+0000 7f4ae27fc640 10 client.4348.objecter ms_dispatch 0x7f4aec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:29.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.316+0000 7f4ae27fc640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:29.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.316+0000 7f4ae27fc640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:29.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.317+0000 7f4ae27fc640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:11:29.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.319+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ae80501b0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.319+0000 7f4ae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:29.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.319+0000 7f4ae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:29.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.569+0000 7f4af419a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.569+0000 7f4af419a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.569+0000 7f4af419a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:29.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.569+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4aec0627b0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.606+0000 7f4ae27fc640 1 -- 172.21.15.46:0/116261575 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f4ae80522b0 con 0x7f4aec0a98f0 2024-03-25T18:11:29.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.607+0000 7f4ae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.607+0000 7f4ae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:29.608 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:29.608 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 10 librados: watch_flush enter 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 10 librados: watch_flush exit 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 >> v1:172.21.15.164:6800/1612084612 conn(0x7f4ad006df70 legacy=0x7f4ad0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 10 monclient: shutdown 2024-03-25T18:11:29.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 >> v1:172.21.15.164:6789/0 conn(0x7f4aec0a98f0 legacy=0x7f4aec0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 shutdown_connections 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 >> 172.21.15.46:0/116261575 conn(0x7f4aec0b7ea0 msgr2=0x7f4aec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 shutdown_connections 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 -- 172.21.15.46:0/116261575 wait complete. 2024-03-25T18:11:29.611 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:29.609+0000 7f4af419a640 1 librados: shutdown 2024-03-25T18:11:29.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:36.428 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.539+0000 7f7d9976d640 10 monclient: get_monmap_and_config 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.539+0000 7f7d9976d640 10 monclient: build_initial_monmap 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.539+0000 7f7d9976d640 10 monclient: monmap: 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:36.540072+0000 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:36.540072+0000 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.540 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:36.541 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:36.541 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:36.541 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.540+0000 7f7d9976d640 1 Processor -- start 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.540+0000 7f7d9976d640 1 -- start start 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.540+0000 7f7d9976d640 10 monclient: init 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.540+0000 7f7d9976d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.540+0000 7f7d9976d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d940a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d9413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d941430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:36.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94142520 con 0x7f7d940a98f0 2024-03-25T18:11:36.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d940b28a0 con 0x7f7d941430c0 2024-03-25T18:11:36.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d940b2a30 con 0x7f7d9413f710 2024-03-25T18:11:36.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:36.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.541+0000 7f7d9976d640 10 monclient(hunting): authenticate will time out at 7405.802700s 2024-03-25T18:11:36.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.553+0000 7f7d92ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7d940a98f0 0x7f7d940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39452/0 (socket says 172.21.15.46:39452) 2024-03-25T18:11:36.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.553+0000 7f7d92ffd640 1 -- 172.21.15.46:0/872599666 learned_addr learned my addr 172.21.15.46:0/872599666 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578978077 0 0) 0x7f7d94142520 con 0x7f7d940a98f0 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.565+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c003520 con 0x7f7d940a98f0 2024-03-25T18:11:36.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433635806 0 0) 0x7f7d940b2a30 con 0x7f7d9413f710 2024-03-25T18:11:36.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:11:36.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.573+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d94142520 con 0x7f7d9413f710 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940771028 0 0) 0x7f7d940b28a0 con 0x7f7d941430c0 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.578+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d940b2a30 con 0x7f7d941430c0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1148798434 0 0) 0x7f7d7c003520 con 0x7f7d940a98f0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 >> v1:172.21.15.46:6790/0 conn(0x7f7d9413f710 legacy=0x7f7d941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 >> v1:172.21.15.164:6789/0 conn(0x7f7d941430c0 legacy=0x7f7d9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 monclient: found mon.noname-a 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d940b44c0 con 0x7f7d940a98f0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7d84001690 con 0x7f7d940a98f0 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d9976d640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d9976d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d91ffb640 10 monclient: dump: 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.585 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:36.586 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:36.586 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:36.586 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.583+0000 7f7d9976d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d840017f0 con 0x7f7d940a98f0 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d9976d640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d9976d640 4 set_mon_vals no callback set 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 1 -- 172.21.15.46:0/872599666 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7d84001c00 con 0x7f7d940a98f0 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.604+0000 7f7d91ffb640 10 monclient: dump: 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.605 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.606 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:36.606 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:36.606 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:36.606 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 10 monclient: shutdown 2024-03-25T18:11:36.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:36.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 1 -- 172.21.15.46:0/872599666 >> v1:172.21.15.46:6789/0 conn(0x7f7d940a98f0 legacy=0x7f7d940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 1 -- 172.21.15.46:0/872599666 shutdown_connections 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 1 -- 172.21.15.46:0/872599666 >> 172.21.15.46:0/872599666 conn(0x7f7d940b7ea0 msgr2=0x7f7d940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 1 -- 172.21.15.46:0/872599666 shutdown_connections 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.605+0000 7f7d9976d640 1 -- 172.21.15.46:0/872599666 wait complete. 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 10 monclient: build_initial_monmap 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 10 monclient: monmap: 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:36.608 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 librados: starting msgr at 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 librados: starting objecter 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 Processor -- start 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 -- start start 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 librados: setting wanted keys 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 1 librados: calling monclient init 2024-03-25T18:11:36.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.606+0000 7f7d9976d640 10 monclient: init 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d940a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d9413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d941430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94132a30 con 0x7f7d940a98f0 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94138c80 con 0x7f7d9413f710 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94138f60 con 0x7f7d941430c0 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:36.610 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.607+0000 7f7d9976d640 10 monclient(hunting): authenticate will time out at 7405.869427s 2024-03-25T18:11:36.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.618+0000 7f7d927fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7d9413f710 0x7f7d94131c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:51016/0 (socket says 172.21.15.46:51016) 2024-03-25T18:11:36.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.618+0000 7f7d927fc640 1 -- 172.21.15.46:0/995465704 learned_addr learned my addr 172.21.15.46:0/995465704 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:36.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.622+0000 7f7d7b7fe640 10 client.?.objecter ms_handle_connect 0x7f7d940a98f0 2024-03-25T18:11:36.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.622+0000 7f7d7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.623+0000 7f7d7b7fe640 10 client.?.objecter ms_handle_connect 0x7f7d9413f710 2024-03-25T18:11:36.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.623+0000 7f7d7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.629 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.629+0000 7f7d7b7fe640 10 client.?.objecter ms_handle_connect 0x7f7d941430c0 2024-03-25T18:11:36.629 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.629+0000 7f7d7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895364439 0 0) 0x7f7d94138c80 con 0x7f7d9413f710 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:11:36.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.633+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d80004eb0 con 0x7f7d9413f710 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111952516 0 0) 0x7f7d94132a30 con 0x7f7d940a98f0 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:11:36.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.638+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d94138c80 con 0x7f7d940a98f0 2024-03-25T18:11:36.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551648854 0 0) 0x7f7d94138f60 con 0x7f7d941430c0 2024-03-25T18:11:36.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:11:36.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.648+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d94132a30 con 0x7f7d941430c0 2024-03-25T18:11:36.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3803508982 0 0) 0x7f7d80004eb0 con 0x7f7d9413f710 2024-03-25T18:11:36.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d94138f60 con 0x7f7d9413f710 2024-03-25T18:11:36.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7d88001a30 con 0x7f7d9413f710 2024-03-25T18:11:36.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.650+0000 7f7d7b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:36.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2538656623 0 0) 0x7f7d94138c80 con 0x7f7d940a98f0 2024-03-25T18:11:36.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d80004eb0 con 0x7f7d940a98f0 2024-03-25T18:11:36.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7d84001b00 con 0x7f7d940a98f0 2024-03-25T18:11:36.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.653+0000 7f7d7b7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:36.654 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080111131 0 0) 0x7f7d80004eb0 con 0x7f7d940a98f0 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 >> v1:172.21.15.46:6790/0 conn(0x7f7d941430c0 legacy=0x7f7d94132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 >> v1:172.21.15.164:6789/0 conn(0x7f7d9413f710 legacy=0x7f7d94131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 1 monclient: found mon.a 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:36.659 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94139240 con 0x7f7d940a98f0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.658+0000 7f7d7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 10 monclient: _renew_subs 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94136e50 con 0x7f7d940a98f0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 10 monclient: _renew_subs 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941389f0 con 0x7f7d940a98f0 2024-03-25T18:11:36.660 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.659+0000 7f7d9976d640 1 librados: init done 2024-03-25T18:11:36.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.660+0000 7f7d9976d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.660+0000 7f7d9976d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.660+0000 7f7d9976d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:36.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.660+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d940627b0 con 0x7f7d940a98f0 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d84001ca0 con 0x7f7d940a98f0 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7d84004c30 con 0x7f7d940a98f0 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d7b7fe640 10 monclient: dump: 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:36.967 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:36.968 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:36.968 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:36.968 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:36.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.966+0000 7f7d917fa640 4 set_mon_vals no callback set 2024-03-25T18:11:36.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.967+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7d8401ac00 con 0x7f7d940a98f0 2024-03-25T18:11:36.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.968+0000 7f7d927fc640 10 monclient: get_auth_request con 0x7f7d8006df70 auth_method 0 2024-03-25T18:11:36.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.968+0000 7f7d7b7fe640 10 client.4350.objecter ms_handle_connect 0x7f7d8006df70 2024-03-25T18:11:36.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7d840547e0 con 0x7f7d940a98f0 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 10 client.4350.objecter ms_dispatch 0x7f7d94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d84050120 con 0x7f7d940a98f0 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:36.974+0000 7f7d7b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:37.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.223+0000 7f7d9976d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.223+0000 7f7d9976d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.223+0000 7f7d9976d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:11:37.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.223+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d9413dd60 con 0x7f7d940a98f0 2024-03-25T18:11:37.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.248+0000 7f7d7b7fe640 1 -- 172.21.15.46:0/995465704 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7d84052220 con 0x7f7d940a98f0 2024-03-25T18:11:37.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.248+0000 7f7d7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.248+0000 7f7d7b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:37.249 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:37.249 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.250+0000 7f7d9976d640 10 librados: watch_flush enter 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.250+0000 7f7d9976d640 10 librados: watch_flush exit 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.250+0000 7f7d9976d640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 >> v1:172.21.15.164:6800/1612084612 conn(0x7f7d8006df70 legacy=0x7f7d80070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 10 monclient: shutdown 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 >> v1:172.21.15.46:6789/0 conn(0x7f7d940a98f0 legacy=0x7f7d94137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 shutdown_connections 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 >> 172.21.15.46:0/995465704 conn(0x7f7d940b7ea0 msgr2=0x7f7d94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 shutdown_connections 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 -- 172.21.15.46:0/995465704 wait complete. 2024-03-25T18:11:37.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:37.251+0000 7f7d9976d640 1 librados: shutdown 2024-03-25T18:11:37.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:38.910 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:14.828 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:38.918 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:14.836 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:38.972 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:14.890 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:38.976 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:14.894 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:38.981 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:14.899 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:38.984 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:14.855 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:39.000 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:14.872 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:39.017 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:14.935 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:39.028 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:14.900 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:44.071 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.191+0000 7f7b21904640 10 monclient: get_monmap_and_config 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.191+0000 7f7b21904640 10 monclient: build_initial_monmap 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.191+0000 7f7b21904640 10 monclient: monmap: 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:44.192317+0000 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:44.192317+0000 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.192+0000 7f7b21904640 1 Processor -- start 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.192+0000 7f7b21904640 1 -- start start 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.192+0000 7f7b21904640 10 monclient: init 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c06ab80 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c06af60 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-c con 0x7f7b1c0af580 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0b04d0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0b1230 con 0x7f7b1c0af580 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0affb0 con 0x7f7b1c06af60 2024-03-25T18:11:44.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:44.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.193+0000 7f7b21904640 10 monclient(hunting): authenticate will time out at 7413.454840s 2024-03-25T18:11:44.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.210+0000 7f7b1b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7b1c0af580 0x7f7b1c14dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:43790/0 (socket says 172.21.15.46:43790) 2024-03-25T18:11:44.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.211+0000 7f7b1b7fe640 1 -- 172.21.15.46:0/1987345306 learned_addr learned my addr 172.21.15.46:0/1987345306 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140566441 0 0) 0x7f7b1c0b1230 con 0x7f7b1c0af580 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:11:44.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.219+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b04003520 con 0x7f7b1c0af580 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.224+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467665275 0 0) 0x7f7b1c0b04d0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.224+0000 7f7b19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.224+0000 7f7b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.224+0000 7f7b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.224+0000 7f7b19ffb640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:11:44.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.225+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1c0b1230 con 0x7f7b1c06ab80 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266106938 0 0) 0x7f7b04003520 con 0x7f7b1c0af580 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 >> v1:172.21.15.46:6790/0 conn(0x7f7b1c06af60 legacy=0x7f7b1c142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 >> v1:172.21.15.46:6789/0 conn(0x7f7b1c06ab80 legacy=0x7f7b1c13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 monclient: found mon.noname-c 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c0b0140 con 0x7f7b1c0af580 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: _finish_auth 0 2024-03-25T18:11:44.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b14001a10 con 0x7f7b1c0af580 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b19ffb640 10 monclient: dump: 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.234 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b21904640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:11:44.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.232+0000 7f7b21904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b14001b70 con 0x7f7b1c0af580 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 1 -- 172.21.15.46:0/1987345306 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b14001f80 con 0x7f7b1c0af580 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b19ffb640 10 monclient: dump: 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.250 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b21904640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.249+0000 7f7b21904640 4 set_mon_vals no callback set 2024-03-25T18:11:44.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 10 monclient: shutdown 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 1 -- 172.21.15.46:0/1987345306 >> v1:172.21.15.164:6789/0 conn(0x7f7b1c0af580 legacy=0x7f7b1c14dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 1 -- 172.21.15.46:0/1987345306 shutdown_connections 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 1 -- 172.21.15.46:0/1987345306 >> 172.21.15.46:0/1987345306 conn(0x7f7b1c0b4b80 msgr2=0x7f7b1c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 1 -- 172.21.15.46:0/1987345306 shutdown_connections 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.250+0000 7f7b21904640 1 -- 172.21.15.46:0/1987345306 wait complete. 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.251+0000 7f7b21904640 10 monclient: build_initial_monmap 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.251+0000 7f7b21904640 10 monclient: monmap: 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:44.254 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.251+0000 7f7b21904640 1 librados: starting msgr at 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.251+0000 7f7b21904640 1 librados: starting objecter 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.251+0000 7f7b21904640 1 Processor -- start 2024-03-25T18:11:44.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.252+0000 7f7b21904640 1 -- start start 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.252+0000 7f7b21904640 1 librados: setting wanted keys 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.252+0000 7f7b21904640 1 librados: calling monclient init 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.252+0000 7f7b21904640 10 monclient: init 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c06ab80 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c0af580 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient(hunting): picked mon.noname-c con 0x7f7b1c11dca0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c11e2f0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c11f0f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c13c9f0 con 0x7f7b1c11dca0 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.253+0000 7f7b21904640 10 monclient(hunting): authenticate will time out at 7413.514818s 2024-03-25T18:11:44.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.259+0000 7f7b1affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b1c06ab80 0x7f7b1c11cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54840/0 (socket says 172.21.15.46:54840) 2024-03-25T18:11:44.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.259+0000 7f7b1affd640 1 -- 172.21.15.46:0/2021336934 learned_addr learned my addr 172.21.15.46:0/2021336934 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:44.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.264+0000 7f7b03fff640 10 client.?.objecter ms_handle_connect 0x7f7b1c06ab80 2024-03-25T18:11:44.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.264+0000 7f7b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.268+0000 7f7b03fff640 10 client.?.objecter ms_handle_connect 0x7f7b1c0af580 2024-03-25T18:11:44.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.268+0000 7f7b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.268+0000 7f7b03fff640 10 client.?.objecter ms_handle_connect 0x7f7b1c11dca0 2024-03-25T18:11:44.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.268+0000 7f7b03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173959407 0 0) 0x7f7b1c11f0f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:11:44.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.274+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08004eb0 con 0x7f7b1c0af580 2024-03-25T18:11:44.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843475322 0 0) 0x7f7b1c13c9f0 con 0x7f7b1c11dca0 2024-03-25T18:11:44.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:11:44.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.285+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1c11f0f0 con 0x7f7b1c11dca0 2024-03-25T18:11:44.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231506552 0 0) 0x7f7b08004eb0 con 0x7f7b1c0af580 2024-03-25T18:11:44.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1c13c9f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b100018c0 con 0x7f7b1c0af580 2024-03-25T18:11:44.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 10 monclient(hunting): dump: 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.294+0000 7f7b03fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765673764 0 0) 0x7f7b1c11e2f0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:11:44.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.297+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08004eb0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3547884500 0 0) 0x7f7b1c11f0f0 con 0x7f7b1c11dca0 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1c11e2f0 con 0x7f7b1c11dca0 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b14001e80 con 0x7f7b1c11dca0 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): dump: 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.313 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762423037 0 0) 0x7f7b08004eb0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1c11f0f0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b0c0019b0 con 0x7f7b1c06ab80 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.312+0000 7f7b03fff640 10 monclient(hunting): dump: 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.314 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.315 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.315 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.315 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.315 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.315 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3957088143 0 0) 0x7f7b1c13c9f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 >> v1:172.21.15.46:6790/0 conn(0x7f7b1c11dca0 legacy=0x7f7b1c144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 >> v1:172.21.15.46:6789/0 conn(0x7f7b1c06ab80 legacy=0x7f7b1c11cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 1 monclient: found mon.b 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c13c3f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 10 monclient: _renew_subs 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c13c640 con 0x7f7b1c0af580 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:44.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 10 monclient: _renew_subs 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c145310 con 0x7f7b1c0af580 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.319+0000 7f7b21904640 1 librados: init done 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.321+0000 7f7b21904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.321+0000 7f7b21904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.321+0000 7f7b21904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.322 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.321+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c11f610 con 0x7f7b1c0af580 2024-03-25T18:11:44.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.596+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b100035f0 con 0x7f7b1c0af580 2024-03-25T18:11:44.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.596+0000 7f7b03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:44.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.597+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7b10003760 con 0x7f7b1c0af580 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.597+0000 7f7b03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.597+0000 7f7b03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.597+0000 7f7b197fa640 4 set_mon_vals no callback set 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.597+0000 7f7b03fff640 10 monclient: dump: 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:44.598 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:44.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.598+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7b10019730 con 0x7f7b1c0af580 2024-03-25T18:11:44.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.598+0000 7f7b1affd640 10 monclient: get_auth_request con 0x7f7b0806e110 auth_method 0 2024-03-25T18:11:44.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.598+0000 7f7b03fff640 10 client.4360.objecter ms_handle_connect 0x7f7b0806e110 2024-03-25T18:11:44.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.602+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7b100533e0 con 0x7f7b1c0af580 2024-03-25T18:11:44.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.602+0000 7f7b03fff640 10 client.4360.objecter ms_dispatch 0x7f7b1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:44.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.602+0000 7f7b03fff640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:44.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.602+0000 7f7b03fff640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:44.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.602+0000 7f7b03fff640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:11:44.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.603+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b1004ed20 con 0x7f7b1c0af580 2024-03-25T18:11:44.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.603+0000 7f7b03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.603+0000 7f7b03fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:44.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.843+0000 7f7b21904640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.843+0000 7f7b21904640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.843+0000 7f7b21904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:44.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.843+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b1c11e0b0 con 0x7f7b1c0af580 2024-03-25T18:11:44.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.863+0000 7f7b03fff640 1 -- 172.21.15.46:0/2021336934 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7b10050e20 con 0x7f7b1c0af580 2024-03-25T18:11:44.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.863+0000 7f7b03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.863+0000 7f7b03fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:44.864 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:44.864 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:44.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 10 librados: watch_flush enter 2024-03-25T18:11:44.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 10 librados: watch_flush exit 2024-03-25T18:11:44.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:11:44.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:11:44.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 >> v1:172.21.15.164:6800/1612084612 conn(0x7f7b0806e110 legacy=0x7f7b080705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 10 monclient: shutdown 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 >> v1:172.21.15.164:6789/0 conn(0x7f7b1c0af580 legacy=0x7f7b1c13bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 shutdown_connections 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 >> 172.21.15.46:0/2021336934 conn(0x7f7b1c0b4b80 msgr2=0x7f7b1c0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.866+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 shutdown_connections 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.867+0000 7f7b21904640 1 -- 172.21.15.46:0/2021336934 wait complete. 2024-03-25T18:11:44.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:44.867+0000 7f7b21904640 1 librados: shutdown 2024-03-25T18:11:44.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:51.686 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:51.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.799+0000 7faf5cdaf640 10 monclient: get_monmap_and_config 2024-03-25T18:11:51.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.799+0000 7faf5cdaf640 10 monclient: build_initial_monmap 2024-03-25T18:11:51.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.799+0000 7faf5cdaf640 10 monclient: monmap: 2024-03-25T18:11:51.801 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:51.801 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:51.801117+0000 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:51.801117+0000 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:51.802 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.800+0000 7faf5cdaf640 1 Processor -- start 2024-03-25T18:11:51.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.800+0000 7faf5cdaf640 1 -- start start 2024-03-25T18:11:51.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.800+0000 7faf5cdaf640 10 monclient: init 2024-03-25T18:11:51.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7faf580a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7faf5813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-b con 0x7faf581430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58142520 con 0x7faf580a98f0 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580b28a0 con 0x7faf5813f710 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580b2a30 con 0x7faf581430c0 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:51.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.801+0000 7faf5cdaf640 10 monclient(hunting): authenticate will time out at 7421.064020s 2024-03-25T18:11:51.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.816+0000 7faf56d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7faf581430c0 0x7faf5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:42356/0 (socket says 172.21.15.46:42356) 2024-03-25T18:11:51.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.816+0000 7faf56d9d640 1 -- 172.21.15.46:0/2652091651 learned_addr learned my addr 172.21.15.46:0/2652091651 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890564568 0 0) 0x7faf580b28a0 con 0x7faf5813f710 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:11:51.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.832+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf44003520 con 0x7faf5813f710 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902367151 0 0) 0x7faf58142520 con 0x7faf580a98f0 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:11:51.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.836+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf580b28a0 con 0x7faf580a98f0 2024-03-25T18:11:51.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.841+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065928546 0 0) 0x7faf44003520 con 0x7faf5813f710 2024-03-25T18:11:51.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 >> v1:172.21.15.46:6790/0 conn(0x7faf581430c0 legacy=0x7faf5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 >> v1:172.21.15.46:6789/0 conn(0x7faf580a98f0 legacy=0x7faf580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 1 monclient: found mon.noname-c 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf580b44c0 con 0x7faf5813f710 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: _finish_auth 0 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7faf40001690 con 0x7faf5813f710 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf5cdaf640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:11:51.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf5cdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf55d9b640 10 monclient: dump: 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.842+0000 7faf5cdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:51.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf400017f0 con 0x7faf5813f710 2024-03-25T18:11:51.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 1 -- 172.21.15.46:0/2652091651 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7faf40001c00 con 0x7faf5813f710 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf5cdaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf5cdaf640 4 set_mon_vals no callback set 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.859+0000 7faf55d9b640 10 monclient: dump: 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:51.861 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:51.862 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 10 monclient: shutdown 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2652091651 >> v1:172.21.15.164:6789/0 conn(0x7faf5813f710 legacy=0x7faf581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2652091651 shutdown_connections 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2652091651 >> 172.21.15.46:0/2652091651 conn(0x7faf580b7ea0 msgr2=0x7faf580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2652091651 shutdown_connections 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.860+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2652091651 wait complete. 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 10 monclient: build_initial_monmap 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 10 monclient: monmap: 2024-03-25T18:11:51.864 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 librados: starting msgr at 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 librados: starting objecter 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 Processor -- start 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 -- start start 2024-03-25T18:11:51.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 librados: setting wanted keys 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 1 librados: calling monclient init 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.861+0000 7faf5cdaf640 10 monclient: init 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-b con 0x7faf580a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7faf5813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7faf581430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580abdc0 con 0x7faf581430c0 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf581d7730 con 0x7faf580a98f0 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58138f10 con 0x7faf5813f710 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:51.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.862+0000 7faf5cdaf640 10 monclient(hunting): authenticate will time out at 7421.124971s 2024-03-25T18:11:51.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.868+0000 7faf56d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faf581430c0 0x7faf581d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59460/0 (socket says 172.21.15.46:59460) 2024-03-25T18:11:51.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.868+0000 7faf56d9d640 1 -- 172.21.15.46:0/2433000343 learned_addr learned my addr 172.21.15.46:0/2433000343 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:51.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.869+0000 7faf4effd640 10 client.?.objecter ms_handle_connect 0x7faf581430c0 2024-03-25T18:11:51.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.869+0000 7faf4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.873+0000 7faf4effd640 10 client.?.objecter ms_handle_connect 0x7faf580a98f0 2024-03-25T18:11:51.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.873+0000 7faf4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.878+0000 7faf4effd640 10 client.?.objecter ms_handle_connect 0x7faf5813f710 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.878+0000 7faf4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563968247 0 0) 0x7faf581d7730 con 0x7faf580a98f0 2024-03-25T18:11:51.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:11:51.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.883+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf38004eb0 con 0x7faf580a98f0 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022512435 0 0) 0x7faf38004eb0 con 0x7faf580a98f0 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf581d7730 con 0x7faf580a98f0 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7faf48001930 con 0x7faf580a98f0 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 10 monclient(hunting): dump: 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.892+0000 7faf4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140131871 0 0) 0x7faf58138f10 con 0x7faf5813f710 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.894+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf38004eb0 con 0x7faf5813f710 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618229144 0 0) 0x7faf580abdc0 con 0x7faf581430c0 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:11:51.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.898+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58138f10 con 0x7faf581430c0 2024-03-25T18:11:51.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820594666 0 0) 0x7faf581d7730 con 0x7faf580a98f0 2024-03-25T18:11:51.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 >> v1:172.21.15.46:6790/0 conn(0x7faf5813f710 legacy=0x7faf581d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 >> v1:172.21.15.46:6789/0 conn(0x7faf581430c0 legacy=0x7faf581d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 1 monclient: found mon.b 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf581390a0 con 0x7faf580a98f0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 10 monclient: _finish_auth 0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 10 monclient: _renew_subs 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf581d8320 con 0x7faf580a98f0 2024-03-25T18:11:51.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 10 monclient: _renew_subs 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf581d43c0 con 0x7faf580a98f0 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.901+0000 7faf5cdaf640 1 librados: init done 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.902+0000 7faf5cdaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:51.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.902+0000 7faf5cdaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:51.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.902+0000 7faf5cdaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:51.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.902+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf580627b0 con 0x7faf580a98f0 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf48003610 con 0x7faf580a98f0 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7faf480039e0 con 0x7faf580a98f0 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf4effd640 10 monclient: dump: 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:51.922 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:51.923 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.920+0000 7faf5559a640 4 set_mon_vals no callback set 2024-03-25T18:11:51.924 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.921+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7faf48003c00 con 0x7faf580a98f0 2024-03-25T18:11:51.924 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.922+0000 7faf4ffff640 10 monclient: get_auth_request con 0x7faf3806de80 auth_method 0 2024-03-25T18:11:51.924 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.922+0000 7faf4effd640 10 client.4366.objecter ms_handle_connect 0x7faf3806de80 2024-03-25T18:11:51.924 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.923+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7faf48053620 con 0x7faf580a98f0 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.923+0000 7faf4effd640 10 client.4366.objecter ms_dispatch 0x7faf58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.923+0000 7faf4effd640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.923+0000 7faf4effd640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:51.923+0000 7faf4effd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T18:11:52.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.177+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf4804f130 con 0x7faf580a98f0 2024-03-25T18:11:52.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.177+0000 7faf4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:52.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.177+0000 7faf4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:52.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.414+0000 7faf5cdaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.414+0000 7faf5cdaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.414+0000 7faf5cdaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:52.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.414+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf58069f40 con 0x7faf580a98f0 2024-03-25T18:11:52.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.455+0000 7faf4effd640 1 -- 172.21.15.46:0/2433000343 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7faf48051060 con 0x7faf580a98f0 2024-03-25T18:11:52.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.455+0000 7faf4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.455+0000 7faf4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:52.457 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:11:52.458 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 10 librados: watch_flush enter 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 10 librados: watch_flush exit 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 10 client.4366.objecter successfully canceled tick 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 >> v1:172.21.15.164:6800/1612084612 conn(0x7faf3806de80 legacy=0x7faf38070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 10 monclient: shutdown 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 >> v1:172.21.15.164:6789/0 conn(0x7faf580a98f0 legacy=0x7faf58138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 shutdown_connections 2024-03-25T18:11:52.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 >> 172.21.15.46:0/2433000343 conn(0x7faf580b7ea0 msgr2=0x7faf58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:52.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 shutdown_connections 2024-03-25T18:11:52.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 -- 172.21.15.46:0/2433000343 wait complete. 2024-03-25T18:11:52.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:52.458+0000 7faf5cdaf640 1 librados: shutdown 2024-03-25T18:11:52.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:58.934 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:34.851 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.979 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:34.850 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.980 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:34.851 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.980 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:34.898 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.981 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:34.899 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.990 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:34.861 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.992 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:34.910 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:58.999 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:34.917 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:59.038 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:34.956 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:59.279 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:59.397 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 10 monclient: get_monmap_and_config 2024-03-25T18:11:59.397 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 10 monclient: build_initial_monmap 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 10 monclient: monmap: 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:11:59.397951+0000 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:11:59.397951+0000 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:11:59.398 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 1 Processor -- start 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 1 -- start start 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.396+0000 7f8c6872b640 10 monclient: init 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c600a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:59.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c6013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c601430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60142520 con 0x7f8c601430c0 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600b28a0 con 0x7f8c6013f710 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600b2a30 con 0x7f8c600a98f0 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:59.400 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.397+0000 7f8c6872b640 10 monclient(hunting): authenticate will time out at 7428.660423s 2024-03-25T18:11:59.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.405+0000 7f8c65cc6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f8c6013f710 0x7f8c601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:43338/0 (socket says 172.21.15.46:43338) 2024-03-25T18:11:59.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.405+0000 7f8c65cc6640 1 -- 172.21.15.46:0/321332272 learned_addr learned my addr 172.21.15.46:0/321332272 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524854288 0 0) 0x7f8c600b28a0 con 0x7f8c6013f710 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:11:59.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.415+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c003520 con 0x7f8c6013f710 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947434518 0 0) 0x7f8c4c003520 con 0x7f8c6013f710 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 >> v1:172.21.15.46:6790/0 conn(0x7f8c600a98f0 legacy=0x7f8c600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 >> v1:172.21.15.46:6789/0 conn(0x7f8c601430c0 legacy=0x7f8c6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 10 monclient: _finish_hunting 0 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 1 monclient: found mon.noname-c 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600b44c0 con 0x7f8c6013f710 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 10 monclient: _finish_auth 0 2024-03-25T18:11:59.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 10 monclient: _check_auth_tickets 2024-03-25T18:11:59.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c654c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:59.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c6872b640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:11:59.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.421+0000 7f8c6872b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c50001690 con 0x7f8c6013f710 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: dump: 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c500018e0 con 0x7f8c6013f710 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 1 -- 172.21.15.46:0/321332272 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c50001c70 con 0x7f8c6013f710 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c654c5640 10 monclient: dump: 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.434 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:59.435 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:59.435 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:59.435 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c6872b640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:59.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.431+0000 7f8c6872b640 4 set_mon_vals no callback set 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 10 monclient: shutdown 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 1 -- 172.21.15.46:0/321332272 >> v1:172.21.15.164:6789/0 conn(0x7f8c6013f710 legacy=0x7f8c601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 1 -- 172.21.15.46:0/321332272 shutdown_connections 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 1 -- 172.21.15.46:0/321332272 >> 172.21.15.46:0/321332272 conn(0x7f8c600b7ea0 msgr2=0x7f8c600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 1 -- 172.21.15.46:0/321332272 shutdown_connections 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.432+0000 7f8c6872b640 1 -- 172.21.15.46:0/321332272 wait complete. 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 10 monclient: build_initial_monmap 2024-03-25T18:11:59.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 10 monclient: monmap: 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 librados: starting msgr at 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 librados: starting objecter 2024-03-25T18:11:59.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 Processor -- start 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 -- start start 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 librados: setting wanted keys 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 1 librados: calling monclient init 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.433+0000 7f8c6872b640 10 monclient: init 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c600a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c6013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c601430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600701a0 con 0x7f8c600a98f0 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60070410 con 0x7f8c601430c0 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6006e950 con 0x7f8c6013f710 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:59.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.434+0000 7f8c6872b640 10 monclient(hunting): authenticate will time out at 7428.696843s 2024-03-25T18:11:59.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.444+0000 7f8c65cc6640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f8c6013f710 0x7f8c60139160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:33622/0 (socket says 172.21.15.46:33622) 2024-03-25T18:11:59.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.444+0000 7f8c65cc6640 1 -- 172.21.15.46:0/107182201 learned_addr learned my addr 172.21.15.46:0/107182201 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:11:59.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.445+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c601430c0 2024-03-25T18:11:59.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.445+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.446+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c6013f710 2024-03-25T18:11:59.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.446+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724263696 0 0) 0x7f8c60070410 con 0x7f8c601430c0 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c44004eb0 con 0x7f8c601430c0 2024-03-25T18:11:59.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148637627 0 0) 0x7f8c6006e950 con 0x7f8c6013f710 2024-03-25T18:11:59.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.451+0000 7f8c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.452+0000 7f8c56ffd640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:11:59.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.452+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60070410 con 0x7f8c6013f710 2024-03-25T18:11:59.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.455+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c600a98f0 2024-03-25T18:11:59.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.455+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1672372591 0 0) 0x7f8c44004eb0 con 0x7f8c601430c0 2024-03-25T18:11:59.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c6006e950 con 0x7f8c601430c0 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c58001b90 con 0x7f8c601430c0 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.463 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.461+0000 7f8c56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:59.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027764686 0 0) 0x7f8c60070410 con 0x7f8c6013f710 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c44004eb0 con 0x7f8c6013f710 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c50001bf0 con 0x7f8c6013f710 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.463+0000 7f8c56ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:59.465 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033188252 0 0) 0x7f8c6006e950 con 0x7f8c601430c0 2024-03-25T18:11:59.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 >> v1:172.21.15.46:6790/0 conn(0x7f8c6013f710 legacy=0x7f8c60139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 >> v1:172.21.15.46:6789/0 conn(0x7f8c600a98f0 legacy=0x7f8c6006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:59.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 1 monclient: found mon.b 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c601d94a0 con 0x7f8c601430c0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 10 monclient: _renew_subs 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c60139870 con 0x7f8c601430c0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 10 monclient: _renew_subs 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6006f360 con 0x7f8c601430c0 2024-03-25T18:11:59.469 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.466+0000 7f8c6872b640 1 librados: init done 2024-03-25T18:11:59.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.467+0000 7f8c6872b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.467+0000 7f8c6872b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.467+0000 7f8c6872b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:11:59.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.467+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c600627b0 con 0x7f8c601430c0 2024-03-25T18:11:59.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c58002080 con 0x7f8c601430c0 2024-03-25T18:11:59.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:59.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8c58003b10 con 0x7f8c601430c0 2024-03-25T18:11:59.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c56ffd640 10 monclient: dump: 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:11:59.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.744+0000 7f8c64cc4640 4 set_mon_vals no callback set 2024-03-25T18:11:59.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.745+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f8c58003d30 con 0x7f8c601430c0 2024-03-25T18:11:59.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.746+0000 7f8c664c7640 10 monclient: get_auth_request con 0x7f8c4406de90 auth_method 0 2024-03-25T18:11:59.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.746+0000 7f8c56ffd640 10 client.4372.objecter ms_handle_connect 0x7f8c4406de90 2024-03-25T18:11:59.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.750+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f8c580538c0 con 0x7f8c601430c0 2024-03-25T18:11:59.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.750+0000 7f8c56ffd640 10 client.4372.objecter ms_dispatch 0x7f8c60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:59.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.750+0000 7f8c56ffd640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:59.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.750+0000 7f8c56ffd640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:59.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.750+0000 7f8c56ffd640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:11:59.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.752+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c5804f200 con 0x7f8c601430c0 2024-03-25T18:11:59.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.752+0000 7f8c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.752+0000 7f8c56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:00.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.999+0000 7f8c6872b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.999+0000 7f8c6872b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.999+0000 7f8c6872b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:00.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:11:59.999+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c60071ec0 con 0x7f8c601430c0 2024-03-25T18:12:00.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.073+0000 7f8c56ffd640 1 -- 172.21.15.46:0/107182201 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f8c58051300 con 0x7f8c601430c0 2024-03-25T18:12:00.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.073+0000 7f8c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.073+0000 7f8c56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:00.075 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:00.075 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 10 librados: watch_flush enter 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 10 librados: watch_flush exit 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 >> v1:172.21.15.164:6800/1612084612 conn(0x7f8c4406de90 legacy=0x7f8c44070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 10 monclient: shutdown 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 >> v1:172.21.15.164:6789/0 conn(0x7f8c601430c0 legacy=0x7f8c601dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:00.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 shutdown_connections 2024-03-25T18:12:00.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 >> 172.21.15.46:0/107182201 conn(0x7f8c600b7ea0 msgr2=0x7f8c60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:00.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 shutdown_connections 2024-03-25T18:12:00.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 -- 172.21.15.46:0/107182201 wait complete. 2024-03-25T18:12:00.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:00.075+0000 7f8c6872b640 1 librados: shutdown 2024-03-25T18:12:00.087 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:06.894 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:07.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.008+0000 7f1482932640 10 monclient: get_monmap_and_config 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.008+0000 7f1482932640 10 monclient: build_initial_monmap 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.008+0000 7f1482932640 10 monclient: monmap: 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:07.009924+0000 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:07.009924+0000 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:07.010 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.009+0000 7f1482932640 1 Processor -- start 2024-03-25T18:12:07.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.009+0000 7f1482932640 1 -- start start 2024-03-25T18:12:07.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.009+0000 7f1482932640 10 monclient: init 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient(hunting): picked mon.noname-a con 0x7f147c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient(hunting): picked mon.noname-c con 0x7f147c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient(hunting): picked mon.noname-b con 0x7f147c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c142520 con 0x7f147c0a98f0 2024-03-25T18:12:07.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c0b28a0 con 0x7f147c13f710 2024-03-25T18:12:07.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c0b2a30 con 0x7f147c1430c0 2024-03-25T18:12:07.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:07.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.010+0000 7f1482932640 10 monclient(hunting): authenticate will time out at 7436.273372s 2024-03-25T18:12:07.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.021+0000 7f1480ecf640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f147c1430c0 0x7f147c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:35984/0 (socket says 172.21.15.46:35984) 2024-03-25T18:12:07.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.021+0000 7f1480ecf640 1 -- 172.21.15.46:0/3840482098 learned_addr learned my addr 172.21.15.46:0/3840482098 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291401172 0 0) 0x7f147c0b2a30 con 0x7f147c1430c0 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:12:07.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.028+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1468003520 con 0x7f147c1430c0 2024-03-25T18:12:07.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187589982 0 0) 0x7f1468003520 con 0x7f147c1430c0 2024-03-25T18:12:07.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 >> v1:172.21.15.164:6789/0 conn(0x7f147c13f710 legacy=0x7f147c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 >> v1:172.21.15.46:6789/0 conn(0x7f147c0a98f0 legacy=0x7f147c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:07.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0b44c0 con 0x7f147c1430c0 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1474001a10 con 0x7f147c1430c0 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f147b7fe640 10 monclient: dump: 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.039 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f1482932640 5 monclient: authenticate success, global_id 4379 2024-03-25T18:12:07.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.036+0000 7f1482932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1474001b70 con 0x7f147c1430c0 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 1 -- 172.21.15.46:0/3840482098 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1474001fc0 con 0x7f147c1430c0 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 4 set_mon_vals no callback set 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:07.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f147b7fe640 10 monclient: dump: 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.055 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 10 monclient: shutdown 2024-03-25T18:12:07.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:07.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 1 -- 172.21.15.46:0/3840482098 >> v1:172.21.15.46:6790/0 conn(0x7f147c1430c0 legacy=0x7f147c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 1 -- 172.21.15.46:0/3840482098 shutdown_connections 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 1 -- 172.21.15.46:0/3840482098 >> 172.21.15.46:0/3840482098 conn(0x7f147c0b7ea0 msgr2=0x7f147c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.053+0000 7f1482932640 1 -- 172.21.15.46:0/3840482098 shutdown_connections 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.054+0000 7f1482932640 1 -- 172.21.15.46:0/3840482098 wait complete. 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.054+0000 7f1482932640 10 monclient: build_initial_monmap 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.054+0000 7f1482932640 10 monclient: monmap: 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:07.057 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 librados: starting msgr at 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 librados: starting objecter 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 Processor -- start 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 -- start start 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 librados: setting wanted keys 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 1 librados: calling monclient init 2024-03-25T18:12:07.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 10 monclient: init 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.055+0000 7f1482932640 10 monclient(hunting): picked mon.noname-a con 0x7f147c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 10 monclient(hunting): picked mon.noname-b con 0x7f147c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 10 monclient(hunting): picked mon.noname-c con 0x7f147c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c1d7500 con 0x7f147c0a98f0 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c1394a0 con 0x7f147c13f710 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c138b50 con 0x7f147c1430c0 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:07.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.056+0000 7f1482932640 10 monclient(hunting): authenticate will time out at 7436.318706s 2024-03-25T18:12:07.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.068+0000 7f147bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f147c0a98f0 0x7f147c138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52102/0 (socket says 172.21.15.46:52102) 2024-03-25T18:12:07.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.069+0000 7f147bfff640 1 -- 172.21.15.46:0/3815501215 learned_addr learned my addr 172.21.15.46:0/3815501215 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:07.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.072+0000 7f14797fa640 10 client.?.objecter ms_handle_connect 0x7f147c13f710 2024-03-25T18:12:07.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.072+0000 7f14797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.073+0000 7f14797fa640 10 client.?.objecter ms_handle_connect 0x7f147c0a98f0 2024-03-25T18:12:07.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.073+0000 7f14797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.080+0000 7f14797fa640 10 client.?.objecter ms_handle_connect 0x7f147c1430c0 2024-03-25T18:12:07.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.080+0000 7f14797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.083 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540677032 0 0) 0x7f147c1d7500 con 0x7f147c0a98f0 2024-03-25T18:12:07.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:12:07.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.082+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c004eb0 con 0x7f147c0a98f0 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236518800 0 0) 0x7f147c1394a0 con 0x7f147c13f710 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:12:07.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.086+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c1d7500 con 0x7f147c13f710 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983341878 0 0) 0x7f147c138b50 con 0x7f147c1430c0 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:12:07.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.095+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c1394a0 con 0x7f147c1430c0 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 631823686 0 0) 0x7f147c1d7500 con 0x7f147c13f710 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147c138b50 con 0x7f147c13f710 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f14640018b0 con 0x7f147c13f710 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 10 monclient(hunting): dump: 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.106 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.107 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.107 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.107 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.107 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.104+0000 7f14797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180448648 0 0) 0x7f147c1394a0 con 0x7f147c1430c0 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147c1d7500 con 0x7f147c1430c0 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1474001e80 con 0x7f147c1430c0 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.115+0000 7f14797fa640 10 monclient(hunting): dump: 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.117 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1427805210 0 0) 0x7f145c004eb0 con 0x7f147c0a98f0 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f147c1394a0 con 0x7f147c0a98f0 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f146c0019d0 con 0x7f147c0a98f0 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.117+0000 7f14797fa640 10 monclient(hunting): dump: 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.119 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2730213223 0 0) 0x7f147c138b50 con 0x7f147c13f710 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 >> v1:172.21.15.46:6790/0 conn(0x7f147c1430c0 legacy=0x7f147c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 >> v1:172.21.15.46:6789/0 conn(0x7f147c0a98f0 legacy=0x7f147c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 1 monclient: found mon.b 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c1d8380 con 0x7f147c13f710 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f14797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f1482932640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f1482932640 10 monclient: _renew_subs 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f1482932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.126+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c139710 con 0x7f147c13f710 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.127+0000 7f1482932640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:07.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.127+0000 7f1482932640 10 monclient: _renew_subs 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.127+0000 7f1482932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.127+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c1d7750 con 0x7f147c13f710 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.127+0000 7f1482932640 1 librados: init done 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.128+0000 7f1482932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.128+0000 7f1482932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.128+0000 7f1482932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:07.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.128+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c0627b0 con 0x7f147c13f710 2024-03-25T18:12:07.402 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14640035f0 con 0x7f147c13f710 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1464003a40 con 0x7f147c13f710 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 10 monclient: dump: 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:07.403 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f147affd640 4 set_mon_vals no callback set 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.401+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f1464003c60 con 0x7f147c13f710 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.402+0000 7f147bfff640 10 monclient: get_auth_request con 0x7f145c06e0f0 auth_method 0 2024-03-25T18:12:07.404 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.402+0000 7f14797fa640 10 client.4378.objecter ms_handle_connect 0x7f145c06e0f0 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f14640535b0 con 0x7f147c13f710 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 10 client.4378.objecter ms_dispatch 0x7f147c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f146404f0c0 con 0x7f147c13f710 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.407+0000 7f14797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:07.638 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.637+0000 7f1482932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.637+0000 7f1482932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.637+0000 7f1482932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:07.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.637+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f147c069f40 con 0x7f147c13f710 2024-03-25T18:12:07.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.681+0000 7f14797fa640 1 -- 172.21.15.46:0/3815501215 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f1464050ff0 con 0x7f147c13f710 2024-03-25T18:12:07.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.681+0000 7f14797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.681+0000 7f14797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:07.685 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:07.685 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 10 librados: watch_flush enter 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 10 librados: watch_flush exit 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 >> v1:172.21.15.164:6800/1612084612 conn(0x7f145c06e0f0 legacy=0x7f145c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 10 monclient: shutdown 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:07.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 >> v1:172.21.15.164:6789/0 conn(0x7f147c13f710 legacy=0x7f147c1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 shutdown_connections 2024-03-25T18:12:07.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 >> 172.21.15.46:0/3815501215 conn(0x7f147c0b7ea0 msgr2=0x7f147c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:07.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 shutdown_connections 2024-03-25T18:12:07.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 -- 172.21.15.46:0/3815501215 wait complete. 2024-03-25T18:12:07.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:07.684+0000 7f1482932640 1 librados: shutdown 2024-03-25T18:12:07.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:14.504 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:14.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 10 monclient: get_monmap_and_config 2024-03-25T18:12:14.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 10 monclient: build_initial_monmap 2024-03-25T18:12:14.616 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 10 monclient: monmap: 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:14.616738+0000 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:14.616738+0000 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:14.617 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 1 Processor -- start 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 1 -- start start 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.615+0000 7ff929939640 10 monclient: init 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient(hunting): picked mon.noname-c con 0x7ff9240a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient(hunting): picked mon.noname-b con 0x7ff92413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient(hunting): picked mon.noname-a con 0x7ff9241430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:14.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff924142520 con 0x7ff9241430c0 2024-03-25T18:12:14.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9240b28a0 con 0x7ff9240a98f0 2024-03-25T18:12:14.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9240b2a30 con 0x7ff92413f710 2024-03-25T18:12:14.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:14.619 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.616+0000 7ff929939640 10 monclient(hunting): authenticate will time out at 7443.879322s 2024-03-25T18:12:14.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.619+0000 7ff922ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff9240a98f0 0x7ff9240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:41288/0 (socket says 172.21.15.46:41288) 2024-03-25T18:12:14.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.619+0000 7ff922ffd640 1 -- 172.21.15.46:0/1106642396 learned_addr learned my addr 172.21.15.46:0/1106642396 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:14.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660178613 0 0) 0x7ff9240b28a0 con 0x7ff9240a98f0 2024-03-25T18:12:14.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:12:14.629 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.627+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff90c003520 con 0x7ff9240a98f0 2024-03-25T18:12:14.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707081364 0 0) 0x7ff90c003520 con 0x7ff9240a98f0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 >> v1:172.21.15.46:6790/0 conn(0x7ff92413f710 legacy=0x7ff9241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 >> v1:172.21.15.46:6789/0 conn(0x7ff9241430c0 legacy=0x7ff92414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 monclient: found mon.noname-c 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9240b44c0 con 0x7ff9240a98f0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff914001690 con 0x7ff9240a98f0 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff921ffb640 10 monclient: dump: 2024-03-25T18:12:14.637 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff929939640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:12:14.638 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.635+0000 7ff929939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9140017f0 con 0x7ff9240a98f0 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 1 -- 172.21.15.46:0/1106642396 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff914001c40 con 0x7ff9240a98f0 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff929939640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff929939640 4 set_mon_vals no callback set 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.644 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff921ffb640 10 monclient: dump: 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:14.645 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff929939640 10 monclient: shutdown 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff929939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.643+0000 7ff929939640 1 -- 172.21.15.46:0/1106642396 >> v1:172.21.15.164:6789/0 conn(0x7ff9240a98f0 legacy=0x7ff9240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.644+0000 7ff929939640 1 -- 172.21.15.46:0/1106642396 shutdown_connections 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.644+0000 7ff929939640 1 -- 172.21.15.46:0/1106642396 >> 172.21.15.46:0/1106642396 conn(0x7ff9240b7ea0 msgr2=0x7ff9240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.644+0000 7ff929939640 1 -- 172.21.15.46:0/1106642396 shutdown_connections 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.644+0000 7ff929939640 1 -- 172.21.15.46:0/1106642396 wait complete. 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 10 monclient: build_initial_monmap 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 10 monclient: monmap: 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:14.647 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 librados: starting msgr at 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 librados: starting objecter 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 Processor -- start 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 -- start start 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 librados: setting wanted keys 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 1 librados: calling monclient init 2024-03-25T18:12:14.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.645+0000 7ff929939640 10 monclient: init 2024-03-25T18:12:14.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:14.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:14.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient(hunting): picked mon.noname-a con 0x7ff9240a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:14.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient(hunting): picked mon.noname-c con 0x7ff92413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:14.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient(hunting): picked mon.noname-b con 0x7ff9241430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:14.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92413be70 con 0x7ff9240a98f0 2024-03-25T18:12:14.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff924131660 con 0x7ff9241430c0 2024-03-25T18:12:14.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9241317f0 con 0x7ff92413f710 2024-03-25T18:12:14.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:14.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.646+0000 7ff929939640 10 monclient(hunting): authenticate will time out at 7443.908913s 2024-03-25T18:12:14.652 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.651+0000 7ff922ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff9240a98f0 0x7ff924130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45042/0 (socket says 172.21.15.46:45042) 2024-03-25T18:12:14.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.651+0000 7ff922ffd640 1 -- 172.21.15.46:0/920536547 learned_addr learned my addr 172.21.15.46:0/920536547 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:14.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.652+0000 7ff90bfff640 10 client.?.objecter ms_handle_connect 0x7ff9240a98f0 2024-03-25T18:12:14.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.652+0000 7ff90bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.665 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294147231 0 0) 0x7ff92413be70 con 0x7ff9240a98f0 2024-03-25T18:12:14.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:12:14.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.664+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff910004eb0 con 0x7ff9240a98f0 2024-03-25T18:12:14.675 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.673+0000 7ff90bfff640 10 client.?.objecter ms_handle_connect 0x7ff9241430c0 2024-03-25T18:12:14.675 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.673+0000 7ff90bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036349624 0 0) 0x7ff910004eb0 con 0x7ff9240a98f0 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff92413be70 con 0x7ff9240a98f0 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff914001bc0 con 0x7ff9240a98f0 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 10 monclient(hunting): dump: 2024-03-25T18:12:14.680 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:14.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.678+0000 7ff90bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:14.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.681+0000 7ff90bfff640 10 client.?.objecter ms_handle_connect 0x7ff92413f710 2024-03-25T18:12:14.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.681+0000 7ff90bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783631112 0 0) 0x7ff924131660 con 0x7ff9241430c0 2024-03-25T18:12:14.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:12:14.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.684+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff910004eb0 con 0x7ff9241430c0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1318386824 0 0) 0x7ff92413be70 con 0x7ff9240a98f0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 >> v1:172.21.15.46:6790/0 conn(0x7ff92413f710 legacy=0x7ff9241d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 >> v1:172.21.15.164:6789/0 conn(0x7ff9241430c0 legacy=0x7ff9241e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 1 monclient: found mon.a 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924131980 con 0x7ff9240a98f0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 10 monclient: _finish_auth 0 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff90bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:14.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 10 monclient: _renew_subs 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff924135c60 con 0x7ff9240a98f0 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 10 monclient: _renew_subs 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9240b7770 con 0x7ff9240a98f0 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.694+0000 7ff929939640 1 librados: init done 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.695+0000 7ff929939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:14.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.695+0000 7ff929939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:14.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.695+0000 7ff929939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:12:14.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:14.695+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9241371d0 con 0x7ff9240a98f0 2024-03-25T18:12:15.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff914004bf0 con 0x7ff9240a98f0 2024-03-25T18:12:15.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:15.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff914004f00 con 0x7ff9240a98f0 2024-03-25T18:12:15.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff90bfff640 10 monclient: dump: 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:15.014 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:15.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.011+0000 7ff9217fa640 4 set_mon_vals no callback set 2024-03-25T18:12:15.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.012+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7ff91401aed0 con 0x7ff9240a98f0 2024-03-25T18:12:15.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.013+0000 7ff9227fc640 10 monclient: get_auth_request con 0x7ff91006dc30 auth_method 0 2024-03-25T18:12:15.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.013+0000 7ff90bfff640 10 client.4380.objecter ms_handle_connect 0x7ff91006dc30 2024-03-25T18:12:15.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7ff914054a20 con 0x7ff9240a98f0 2024-03-25T18:12:15.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 10 client.4380.objecter ms_dispatch 0x7ff924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:15.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:15.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:15.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-25T18:12:15.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff914050360 con 0x7ff9240a98f0 2024-03-25T18:12:15.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:15.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.020+0000 7ff90bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:15.261 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.260+0000 7ff929939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.260+0000 7ff929939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.260+0000 7ff929939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:12:15.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.260+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9240627b0 con 0x7ff9240a98f0 2024-03-25T18:12:15.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.284+0000 7ff90bfff640 1 -- 172.21.15.46:0/920536547 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7ff914052460 con 0x7ff9240a98f0 2024-03-25T18:12:15.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.284+0000 7ff90bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.284+0000 7ff90bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:15.286 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:15.286 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:15.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 10 librados: watch_flush enter 2024-03-25T18:12:15.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 10 librados: watch_flush exit 2024-03-25T18:12:15.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 10 client.4380.objecter successfully canceled tick 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 >> v1:172.21.15.164:6800/1612084612 conn(0x7ff91006dc30 legacy=0x7ff9100700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 10 monclient: shutdown 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 >> v1:172.21.15.46:6789/0 conn(0x7ff9240a98f0 legacy=0x7ff924130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 shutdown_connections 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.286+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 >> 172.21.15.46:0/920536547 conn(0x7ff9240b7ea0 msgr2=0x7ff924074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.287+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 shutdown_connections 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.287+0000 7ff929939640 1 -- 172.21.15.46:0/920536547 wait complete. 2024-03-25T18:12:15.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:15.287+0000 7ff929939640 1 librados: shutdown 2024-03-25T18:12:15.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:19.106 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:55.024 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.112 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:55.029 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.154 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:03:55.072 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.162 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:55.079 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.167 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:55.038 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.171 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:55.042 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.175 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:55.092 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.206 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:03:55.077 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:19.210 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:03:55.128 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:22.107 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:22.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 10 monclient: get_monmap_and_config 2024-03-25T18:12:22.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 10 monclient: build_initial_monmap 2024-03-25T18:12:22.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 10 monclient: monmap: 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:22.220730+0000 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:22.220730+0000 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:22.221 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 1 Processor -- start 2024-03-25T18:12:22.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 1 -- start start 2024-03-25T18:12:22.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.219+0000 7fcb8d972640 10 monclient: init 2024-03-25T18:12:22.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-a con 0x7fcb880a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-b con 0x7fcb881430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88142520 con 0x7fcb880a98f0 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb880b28a0 con 0x7fcb8813f710 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb880b2a30 con 0x7fcb881430c0 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:22.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.220+0000 7fcb8d972640 10 monclient(hunting): authenticate will time out at 7451.483381s 2024-03-25T18:12:22.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.228+0000 7fcb7ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcb8813f710 0x7fcb881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45828/0 (socket says 172.21.15.46:45828) 2024-03-25T18:12:22.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.228+0000 7fcb7ffff640 1 -- 172.21.15.46:0/1334436060 learned_addr learned my addr 172.21.15.46:0/1334436060 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666367999 0 0) 0x7fcb880b28a0 con 0x7fcb8813f710 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:12:22.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.242+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb74003520 con 0x7fcb8813f710 2024-03-25T18:12:22.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153457453 0 0) 0x7fcb88142520 con 0x7fcb880a98f0 2024-03-25T18:12:22.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:12:22.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.263+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb880b28a0 con 0x7fcb880a98f0 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047812539 0 0) 0x7fcb880b2a30 con 0x7fcb881430c0 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:12:22.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.271+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb88142520 con 0x7fcb881430c0 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335906163 0 0) 0x7fcb74003520 con 0x7fcb8813f710 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 >> v1:172.21.15.46:6790/0 conn(0x7fcb881430c0 legacy=0x7fcb8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 >> v1:172.21.15.46:6789/0 conn(0x7fcb880a98f0 legacy=0x7fcb880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 monclient: found mon.noname-c 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880b44c0 con 0x7fcb8813f710 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:22.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb70001690 con 0x7fcb8813f710 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb8d972640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb8d972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb867fc640 10 monclient: dump: 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.281 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.282 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.282 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.278+0000 7fcb8d972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:22.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb700017f0 con 0x7fcb8813f710 2024-03-25T18:12:22.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:22.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 1 -- 172.21.15.46:0/1334436060 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb70001c00 con 0x7fcb8813f710 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb867fc640 10 monclient: dump: 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb8d972640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:22.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.288+0000 7fcb8d972640 4 set_mon_vals no callback set 2024-03-25T18:12:22.292 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 10 monclient: shutdown 2024-03-25T18:12:22.292 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:22.292 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 1 -- 172.21.15.46:0/1334436060 >> v1:172.21.15.164:6789/0 conn(0x7fcb8813f710 legacy=0x7fcb881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 1 -- 172.21.15.46:0/1334436060 shutdown_connections 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 1 -- 172.21.15.46:0/1334436060 >> 172.21.15.46:0/1334436060 conn(0x7fcb880b7ea0 msgr2=0x7fcb880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 1 -- 172.21.15.46:0/1334436060 shutdown_connections 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.289+0000 7fcb8d972640 1 -- 172.21.15.46:0/1334436060 wait complete. 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 10 monclient: build_initial_monmap 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 10 monclient: monmap: 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:22.293 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 librados: starting msgr at 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 librados: starting objecter 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 Processor -- start 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 -- start start 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 librados: setting wanted keys 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 1 librados: calling monclient init 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.290+0000 7fcb8d972640 10 monclient: init 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-a con 0x7fcb880a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient(hunting): picked mon.noname-b con 0x7fcb881430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88138c10 con 0x7fcb880a98f0 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88138e60 con 0x7fcb881430c0 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88139140 con 0x7fcb8813f710 2024-03-25T18:12:22.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:22.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.291+0000 7fcb8d972640 10 monclient(hunting): authenticate will time out at 7451.553835s 2024-03-25T18:12:22.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.294+0000 7fcb877fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcb881430c0 0x7fcb881e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45910/0 (socket says 172.21.15.46:45910) 2024-03-25T18:12:22.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.294+0000 7fcb877fe640 1 -- 172.21.15.46:0/2572408677 learned_addr learned my addr 172.21.15.46:0/2572408677 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:22.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.295+0000 7fcb84ff9640 10 client.?.objecter ms_handle_connect 0x7fcb881430c0 2024-03-25T18:12:22.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.295+0000 7fcb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.299 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.298+0000 7fcb84ff9640 10 client.?.objecter ms_handle_connect 0x7fcb880a98f0 2024-03-25T18:12:22.299 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.298+0000 7fcb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.352 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.351+0000 7fcb84ff9640 10 client.?.objecter ms_handle_connect 0x7fcb8813f710 2024-03-25T18:12:22.352 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.351+0000 7fcb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768675856 0 0) 0x7fcb88138c10 con 0x7fcb880a98f0 2024-03-25T18:12:22.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:12:22.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.368+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb68004eb0 con 0x7fcb880a98f0 2024-03-25T18:12:22.374 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.372+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158394433 0 0) 0x7fcb88139140 con 0x7fcb8813f710 2024-03-25T18:12:22.374 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.372+0000 7fcb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.374 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.372+0000 7fcb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.374 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.372+0000 7fcb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.374 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.372+0000 7fcb84ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:12:22.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.373+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb88138c10 con 0x7fcb8813f710 2024-03-25T18:12:22.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625470076 0 0) 0x7fcb88138e60 con 0x7fcb881430c0 2024-03-25T18:12:22.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:12:22.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.386+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb88139140 con 0x7fcb881430c0 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058356680 0 0) 0x7fcb88138c10 con 0x7fcb8813f710 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb88138e60 con 0x7fcb8813f710 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb70001b00 con 0x7fcb8813f710 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:22.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.393+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7084500 0 0) 0x7fcb88139140 con 0x7fcb881430c0 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.394+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb88138c10 con 0x7fcb881430c0 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.394+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb80001bd0 con 0x7fcb881430c0 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.394+0000 7fcb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.394+0000 7fcb84ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.394+0000 7fcb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.397 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163159929 0 0) 0x7fcb68004eb0 con 0x7fcb880a98f0 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb88139140 con 0x7fcb880a98f0 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb78001970 con 0x7fcb880a98f0 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.398+0000 7fcb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.399 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.400 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.400 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.400 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2635108924 0 0) 0x7fcb88138c10 con 0x7fcb881430c0 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 >> v1:172.21.15.46:6790/0 conn(0x7fcb8813f710 legacy=0x7fcb88132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 >> v1:172.21.15.46:6789/0 conn(0x7fcb880a98f0 legacy=0x7fcb88137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 1 monclient: found mon.b 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88139920 con 0x7fcb881430c0 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:12:22.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 10 monclient: _renew_subs 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb881382e0 con 0x7fcb881430c0 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 10 monclient: _renew_subs 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb880ab8b0 con 0x7fcb881430c0 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.409+0000 7fcb8d972640 1 librados: init done 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.411+0000 7fcb8d972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.411+0000 7fcb8d972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.411+0000 7fcb8d972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.411+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8813d010 con 0x7fcb881430c0 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb800020a0 con 0x7fcb881430c0 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb80003b30 con 0x7fcb881430c0 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb84ff9640 10 monclient: dump: 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:22.683 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.682+0000 7fcb85ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.683+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fcb80003d50 con 0x7fcb881430c0 2024-03-25T18:12:22.684 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.683+0000 7fcb86ffd640 10 monclient: get_auth_request con 0x7fcb6806e110 auth_method 0 2024-03-25T18:12:22.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.683+0000 7fcb84ff9640 10 client.4390.objecter ms_handle_connect 0x7fcb6806e110 2024-03-25T18:12:22.690 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.690+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fcb80054a00 con 0x7fcb881430c0 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.690+0000 7fcb84ff9640 10 client.4390.objecter ms_dispatch 0x7fcb88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.690+0000 7fcb84ff9640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.690+0000 7fcb84ff9640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.690+0000 7fcb84ff9640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.691+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb80050510 con 0x7fcb881430c0 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.691+0000 7fcb84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.691 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.691+0000 7fcb84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:22.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.945+0000 7fcb8d972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:22.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.945+0000 7fcb8d972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:22.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.945+0000 7fcb8d972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:22.946 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.945+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb88139660 con 0x7fcb881430c0 2024-03-25T18:12:22.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.994+0000 7fcb84ff9640 1 -- 172.21.15.46:0/2572408677 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fcb88139660 con 0x7fcb881430c0 2024-03-25T18:12:22.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.994+0000 7fcb84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:22.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.994+0000 7fcb84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:22.995 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:22.995 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.996+0000 7fcb8d972640 10 librados: watch_flush enter 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 10 librados: watch_flush exit 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 10 client.4390.objecter successfully canceled tick 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 >> v1:172.21.15.164:6800/1612084612 conn(0x7fcb6806e110 legacy=0x7fcb680705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 10 monclient: shutdown 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 >> v1:172.21.15.164:6789/0 conn(0x7fcb881430c0 legacy=0x7fcb881e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 shutdown_connections 2024-03-25T18:12:22.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 >> 172.21.15.46:0/2572408677 conn(0x7fcb880b7ea0 msgr2=0x7fcb88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:22.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 shutdown_connections 2024-03-25T18:12:22.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 -- 172.21.15.46:0/2572408677 wait complete. 2024-03-25T18:12:22.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:22.997+0000 7fcb8d972640 1 librados: shutdown 2024-03-25T18:12:23.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:29.816 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.930+0000 7f5f9df46640 10 monclient: get_monmap_and_config 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.930+0000 7f5f9df46640 10 monclient: build_initial_monmap 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.930+0000 7f5f9df46640 10 monclient: monmap: 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:29.931344+0000 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:29.931344+0000 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:29.931 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:29.932 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:29.932 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:29.932 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:29.932 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.931+0000 7f5f9df46640 1 Processor -- start 2024-03-25T18:12:29.932 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.931+0000 7f5f9df46640 1 -- start start 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.931+0000 7f5f9df46640 10 monclient: init 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-c con 0x7f5f980a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-b con 0x7f5f981430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98142520 con 0x7f5f9813f710 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b28a0 con 0x7f5f980a98f0 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b2a30 con 0x7f5f981430c0 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:29.933 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.932+0000 7f5f9df46640 10 monclient(hunting): authenticate will time out at 7459.194077s 2024-03-25T18:12:29.940 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.940+0000 7f5f96ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f9813f710 0x7f5f981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34016/0 (socket says 172.21.15.46:34016) 2024-03-25T18:12:29.941 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.940+0000 7f5f96ffd640 1 -- 172.21.15.46:0/513814481 learned_addr learned my addr 172.21.15.46:0/513814481 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:29.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803243726 0 0) 0x7f5f980b2a30 con 0x7f5f981430c0 2024-03-25T18:12:29.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:29.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:29.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:29.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:12:29.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.951+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f80003520 con 0x7f5f981430c0 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684297287 0 0) 0x7f5f980b28a0 con 0x7f5f980a98f0 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f980b2a30 con 0x7f5f980a98f0 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985138556 0 0) 0x7f5f98142520 con 0x7f5f9813f710 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:29.960 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:29.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:12:29.961 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.959+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f980b28a0 con 0x7f5f9813f710 2024-03-25T18:12:29.968 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718006433 0 0) 0x7f5f80003520 con 0x7f5f981430c0 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 >> v1:172.21.15.164:6789/0 conn(0x7f5f980a98f0 legacy=0x7f5f980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 >> v1:172.21.15.46:6789/0 conn(0x7f5f9813f710 legacy=0x7f5f981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 1 monclient: found mon.noname-b 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.967+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980b44c0 con 0x7f5f981430c0 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:29.969 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5f90001a10 con 0x7f5f981430c0 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f9df46640 5 monclient: authenticate success, global_id 4397 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f9df46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f967fc640 10 monclient: dump: 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:29.970 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:29.971 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:29.971 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:29.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.968+0000 7f5f9df46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:29.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f90001c30 con 0x7f5f981430c0 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 1 -- 172.21.15.46:0/513814481 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5f90001fc0 con 0x7f5f981430c0 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f967fc640 10 monclient: dump: 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:29.991 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:29.992 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:29.992 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:29.992 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:29.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f9df46640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:29.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.990+0000 7f5f9df46640 4 set_mon_vals no callback set 2024-03-25T18:12:29.993 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 10 monclient: shutdown 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 1 -- 172.21.15.46:0/513814481 >> v1:172.21.15.46:6790/0 conn(0x7f5f981430c0 legacy=0x7f5f9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 1 -- 172.21.15.46:0/513814481 shutdown_connections 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 1 -- 172.21.15.46:0/513814481 >> 172.21.15.46:0/513814481 conn(0x7f5f980b7ea0 msgr2=0x7f5f980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 1 -- 172.21.15.46:0/513814481 shutdown_connections 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 1 -- 172.21.15.46:0/513814481 wait complete. 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.991+0000 7f5f9df46640 10 monclient: build_initial_monmap 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 10 monclient: monmap: 2024-03-25T18:12:29.994 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 librados: starting msgr at 2024-03-25T18:12:29.995 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 librados: starting objecter 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 Processor -- start 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 -- start start 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 librados: setting wanted keys 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 1 librados: calling monclient init 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.992+0000 7f5f9df46640 10 monclient: init 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-a con 0x7f5f980a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient(hunting): picked mon.noname-c con 0x7f5f981430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9813c9b0 con 0x7f5f980a98f0 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f981397c0 con 0x7f5f9813f710 2024-03-25T18:12:29.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98139950 con 0x7f5f981430c0 2024-03-25T18:12:29.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:29.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.993+0000 7f5f9df46640 10 monclient(hunting): authenticate will time out at 7459.254991s 2024-03-25T18:12:29.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.997+0000 7f5f977fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f980a98f0 0x7f5f98130f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34076/0 (socket says 172.21.15.46:34076) 2024-03-25T18:12:29.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.997+0000 7f5f977fe640 1 -- 172.21.15.46:0/3976090555 learned_addr learned my addr 172.21.15.46:0/3976090555 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:29.999 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.999+0000 7f5f7ffff640 10 client.?.objecter ms_handle_connect 0x7f5f980a98f0 2024-03-25T18:12:30.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:29.999+0000 7f5f7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.002+0000 7f5f7ffff640 10 client.?.objecter ms_handle_connect 0x7f5f981430c0 2024-03-25T18:12:30.002 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.002+0000 7f5f7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766982087 0 0) 0x7f5f9813c9b0 con 0x7f5f980a98f0 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:12:30.005 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.004+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f84004eb0 con 0x7f5f980a98f0 2024-03-25T18:12:30.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.007+0000 7f5f7ffff640 10 client.?.objecter ms_handle_connect 0x7f5f9813f710 2024-03-25T18:12:30.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.007+0000 7f5f7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f96ffd640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6789/0 conn(0x7f5f9813f710 legacy=0x7f5f981dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f96ffd640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6789/0 conn(0x7f5f9813f710 legacy=0x7f5f981dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f96ffd640 1 --1- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6789/0 conn(0x7f5f9813f710 0x7f5f981dae50 :-1 s=OPENED pgs=24170 cs=1 l=1).handle_message read tag failed 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f96ffd640 1 --1- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6789/0 conn(0x7f5f9813f710 0x7f5f981dae50 :-1 s=OPENED pgs=24170 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.164:6789/0 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227348557 0 0) 0x7f5f98139950 con 0x7f5f981430c0 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:12:30.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.009+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9813c9b0 con 0x7f5f981430c0 2024-03-25T18:12:30.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504290919 0 0) 0x7f5f9813c9b0 con 0x7f5f981430c0 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f98139950 con 0x7f5f981430c0 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5f90001da0 con 0x7f5f981430c0 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 10 monclient(hunting): dump: 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:30.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.022+0000 7f5f7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3999645221 0 0) 0x7f5f84004eb0 con 0x7f5f980a98f0 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f9813c9b0 con 0x7f5f980a98f0 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5f88001a30 con 0x7f5f980a98f0 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:30.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.030+0000 7f5f7ffff640 10 monclient(hunting): dump: 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:30.032 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:30.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.035+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291323635 0 0) 0x7f5f98139950 con 0x7f5f981430c0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6789/0 conn(0x7f5f9813f710 legacy=0x7f5f981dae50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.46:6789/0 conn(0x7f5f980a98f0 legacy=0x7f5f98130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 1 monclient: found mon.c 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98131680 con 0x7f5f981430c0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 10 monclient: _renew_subs 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f98131950 con 0x7f5f981430c0 2024-03-25T18:12:30.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 10 monclient: _renew_subs 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f98135a30 con 0x7f5f981430c0 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.036+0000 7f5f9df46640 1 librados: init done 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.037+0000 7f5f9df46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.037+0000 7f5f9df46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.037+0000 7f5f9df46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:12:30.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.037+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9813c810 con 0x7f5f981430c0 2024-03-25T18:12:30.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.256+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f90003970 con 0x7f5f981430c0 2024-03-25T18:12:30.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.256+0000 7f5f7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:30.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.256+0000 7f5f95ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.353+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f5f90003ae0 con 0x7f5f981430c0 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.353+0000 7f5f7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.353+0000 7f5f7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.353+0000 7f5f7ffff640 10 monclient: dump: 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:30.353 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:30.354 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:30.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.353+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f5f90019ab0 con 0x7f5f981430c0 2024-03-25T18:12:30.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.354+0000 7f5f977fe640 10 monclient: get_auth_request con 0x7f5f8406de40 auth_method 0 2024-03-25T18:12:30.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.354+0000 7f5f7ffff640 10 client.4400.objecter ms_handle_connect 0x7f5f8406de40 2024-03-25T18:12:30.360 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f5f90053660 con 0x7f5f981430c0 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 10 client.4400.objecter ms_dispatch 0x7f5f98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f9004efa0 con 0x7f5f981430c0 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.361 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.360+0000 7f5f7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:30.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.608+0000 7f5f9df46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.608+0000 7f5f9df46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.608+0000 7f5f9df46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:12:30.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.608+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f98136fc0 con 0x7f5f981430c0 2024-03-25T18:12:30.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.632+0000 7f5f7ffff640 1 -- 172.21.15.46:0/3976090555 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f5f900510a0 con 0x7f5f981430c0 2024-03-25T18:12:30.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.632+0000 7f5f7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.632 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.632+0000 7f5f7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:30.633 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:30.633 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 10 librados: watch_flush enter 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 10 librados: watch_flush exit 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 10 client.4400.objecter successfully canceled tick 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.164:6800/1612084612 conn(0x7f5f8406de40 legacy=0x7f5f84070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 10 monclient: shutdown 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 >> v1:172.21.15.46:6790/0 conn(0x7f5f981430c0 legacy=0x7f5f981e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 shutdown_connections 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.634+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 >> 172.21.15.46:0/3976090555 conn(0x7f5f980b7ea0 msgr2=0x7f5f9814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.635+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 shutdown_connections 2024-03-25T18:12:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.635+0000 7f5f9df46640 1 -- 172.21.15.46:0/3976090555 wait complete. 2024-03-25T18:12:30.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:30.635+0000 7f5f9df46640 1 librados: shutdown 2024-03-25T18:12:30.648 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:37.454 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:37.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.568+0000 7f9173c2a640 10 monclient: get_monmap_and_config 2024-03-25T18:12:37.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.568+0000 7f9173c2a640 10 monclient: build_initial_monmap 2024-03-25T18:12:37.569 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.568+0000 7f9173c2a640 10 monclient: monmap: 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:37.569635+0000 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:37.569635+0000 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:37.570 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.569+0000 7f9173c2a640 1 Processor -- start 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 1 -- start start 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 10 monclient: init 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f916c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.570+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f916c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f916c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c142520 con 0x7f916c1430c0 2024-03-25T18:12:37.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0b28a0 con 0x7f916c13f710 2024-03-25T18:12:37.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0b2a30 con 0x7f916c0a98f0 2024-03-25T18:12:37.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:37.573 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.571+0000 7f9173c2a640 10 monclient(hunting): authenticate will time out at 7466.832897s 2024-03-25T18:12:37.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.587+0000 7f91711c5640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f916c13f710 0x7f916c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36056/0 (socket says 172.21.15.46:36056) 2024-03-25T18:12:37.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.587+0000 7f91711c5640 1 -- 172.21.15.46:0/3426276695 learned_addr learned my addr 172.21.15.46:0/3426276695 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868181257 0 0) 0x7f916c0b28a0 con 0x7f916c13f710 2024-03-25T18:12:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:12:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.594+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9158003520 con 0x7f916c13f710 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899670906 0 0) 0x7f916c142520 con 0x7f916c1430c0 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:12:37.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.602+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c0b28a0 con 0x7f916c1430c0 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470630589 0 0) 0x7f9158003520 con 0x7f916c13f710 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 >> v1:172.21.15.46:6790/0 conn(0x7f916c0a98f0 legacy=0x7f916c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 >> v1:172.21.15.46:6789/0 conn(0x7f916c1430c0 legacy=0x7f916c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: _finish_hunting 0 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 monclient: found mon.noname-c 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0b44c0 con 0x7f916c13f710 2024-03-25T18:12:37.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: _finish_auth 0 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: _check_auth_tickets 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f915c001690 con 0x7f916c13f710 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f91709c4640 10 monclient: dump: 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:37.613 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:37.614 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:37.614 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:37.614 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f9173c2a640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:12:37.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.611+0000 7f9173c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:37.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f915c0017f0 con 0x7f916c13f710 2024-03-25T18:12:37.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:37.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 1 -- 172.21.15.46:0/3426276695 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f915c001c00 con 0x7f916c13f710 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f9173c2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f9173c2a640 4 set_mon_vals no callback set 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.617+0000 7f91709c4640 10 monclient: dump: 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.618 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:37.619 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:37.619 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:37.619 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:37.619 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 10 monclient: shutdown 2024-03-25T18:12:37.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:37.620 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 1 -- 172.21.15.46:0/3426276695 >> v1:172.21.15.164:6789/0 conn(0x7f916c13f710 legacy=0x7f916c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 1 -- 172.21.15.46:0/3426276695 shutdown_connections 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 1 -- 172.21.15.46:0/3426276695 >> 172.21.15.46:0/3426276695 conn(0x7f916c0b7ea0 msgr2=0x7f916c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 1 -- 172.21.15.46:0/3426276695 shutdown_connections 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.618+0000 7f9173c2a640 1 -- 172.21.15.46:0/3426276695 wait complete. 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.619+0000 7f9173c2a640 10 monclient: build_initial_monmap 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.619+0000 7f9173c2a640 10 monclient: monmap: 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:37.621 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 librados: starting msgr at 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 librados: starting objecter 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 Processor -- start 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 -- start start 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 librados: setting wanted keys 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 librados: calling monclient init 2024-03-25T18:12:37.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient: init 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f916c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f916c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f916c1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c138a50 con 0x7f916c0a98f0 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0ab9e0 con 0x7f916c1430c0 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c1d7880 con 0x7f916c13f710 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:37.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.620+0000 7f9173c2a640 10 monclient(hunting): authenticate will time out at 7466.882465s 2024-03-25T18:12:37.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.626+0000 7f91721c7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f916c1430c0 0x7f916c1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36102/0 (socket says 172.21.15.46:36102) 2024-03-25T18:12:37.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.626+0000 7f91721c7640 1 -- 172.21.15.46:0/4220070447 learned_addr learned my addr 172.21.15.46:0/4220070447 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:37.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.627+0000 7f91627fc640 10 client.?.objecter ms_handle_connect 0x7f916c1430c0 2024-03-25T18:12:37.628 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.627+0000 7f91627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764378885 0 0) 0x7f916c0ab9e0 con 0x7f916c1430c0 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:12:37.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.635+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150004eb0 con 0x7f916c1430c0 2024-03-25T18:12:37.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.647+0000 7f91627fc640 10 client.?.objecter ms_handle_connect 0x7f916c13f710 2024-03-25T18:12:37.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.647+0000 7f91627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.653+0000 7f91627fc640 10 client.?.objecter ms_handle_connect 0x7f916c0a98f0 2024-03-25T18:12:37.654 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.653+0000 7f91627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 621779867 0 0) 0x7f9150004eb0 con 0x7f916c1430c0 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f916c0ab9e0 con 0x7f916c1430c0 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9164001c90 con 0x7f916c1430c0 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 10 monclient(hunting): dump: 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.656 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.655+0000 7f91627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:37.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697311705 0 0) 0x7f916c1d7880 con 0x7f916c13f710 2024-03-25T18:12:37.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.663 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.663 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.663 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:12:37.663 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.662+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150004eb0 con 0x7f916c13f710 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461691405 0 0) 0x7f916c0ab9e0 con 0x7f916c1430c0 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 >> v1:172.21.15.46:6790/0 conn(0x7f916c13f710 legacy=0x7f916c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 >> v1:172.21.15.46:6789/0 conn(0x7f916c0a98f0 legacy=0x7f916c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 1 monclient: found mon.b 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c1d84b0 con 0x7f916c1430c0 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f91627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 10 monclient: _renew_subs 2024-03-25T18:12:37.672 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c1d51f0 con 0x7f916c1430c0 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 10 monclient: _renew_subs 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c138e60 con 0x7f916c1430c0 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.671+0000 7f9173c2a640 1 librados: init done 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.672+0000 7f9173c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.672+0000 7f9173c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.672+0000 7f9173c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:37.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.672+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c1d76e0 con 0x7f916c1430c0 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9164003990 con 0x7f916c1430c0 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9164003c00 con 0x7f916c1430c0 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:37.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.936+0000 7f91627fc640 10 monclient: dump: 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:37.938 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:37.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.937+0000 7f9163fff640 4 set_mon_vals no callback set 2024-03-25T18:12:37.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.938+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f9164003e20 con 0x7f916c1430c0 2024-03-25T18:12:37.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.939+0000 7f91719c6640 10 monclient: get_auth_request con 0x7f915006dd90 auth_method 0 2024-03-25T18:12:37.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.939+0000 7f91627fc640 10 client.4399.objecter ms_handle_connect 0x7f915006dd90 2024-03-25T18:12:37.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.941+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f9164054920 con 0x7f916c1430c0 2024-03-25T18:12:37.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.941+0000 7f91627fc640 10 client.4399.objecter ms_dispatch 0x7f916c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:37.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.941+0000 7f91627fc640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:37.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.941+0000 7f91627fc640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:37.942 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.942+0000 7f91627fc640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:12:37.944 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.944+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9164050430 con 0x7f916c1430c0 2024-03-25T18:12:37.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.944+0000 7f91627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.945 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:37.944+0000 7f91627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:38.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.202+0000 7f9173c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.202+0000 7f9173c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.202+0000 7f9173c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:38.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.202+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f916c0627b0 con 0x7f916c1430c0 2024-03-25T18:12:38.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.228+0000 7f91627fc640 1 -- 172.21.15.46:0/4220070447 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f9164052360 con 0x7f916c1430c0 2024-03-25T18:12:38.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.228+0000 7f91627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.228+0000 7f91627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:38.229 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:38.230 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:38.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 10 librados: watch_flush enter 2024-03-25T18:12:38.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 10 librados: watch_flush exit 2024-03-25T18:12:38.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 >> v1:172.21.15.164:6800/1612084612 conn(0x7f915006dd90 legacy=0x7f9150070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 10 monclient: shutdown 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 >> v1:172.21.15.164:6789/0 conn(0x7f916c1430c0 legacy=0x7f916c1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 shutdown_connections 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 >> 172.21.15.46:0/4220070447 conn(0x7f916c0b7ea0 msgr2=0x7f916c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 shutdown_connections 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 -- 172.21.15.46:0/4220070447 wait complete. 2024-03-25T18:12:38.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:38.231+0000 7f9173c2a640 1 librados: shutdown 2024-03-25T18:12:38.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:45.051 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.165+0000 7f18f203e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.165+0000 7f18f203e640 10 monclient: build_initial_monmap 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.165+0000 7f18f203e640 10 monclient: monmap: 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:45.166265+0000 2024-03-25T18:12:45.166 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:45.166265+0000 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:45.167 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.166+0000 7f18f203e640 1 Processor -- start 2024-03-25T18:12:45.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.166+0000 7f18f203e640 1 -- start start 2024-03-25T18:12:45.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.166+0000 7f18f203e640 10 monclient: init 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-a con 0x7f18ec0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-b con 0x7f18ec13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-c con 0x7f18ec1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec142520 con 0x7f18ec0a98f0 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec0b28a0 con 0x7f18ec1430c0 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec0b2a30 con 0x7f18ec13f710 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:45.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.167+0000 7f18f203e640 10 monclient(hunting): authenticate will time out at 7474.428987s 2024-03-25T18:12:45.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.186+0000 7f18eb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f18ec0a98f0 0x7f18ec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36726/0 (socket says 172.21.15.46:36726) 2024-03-25T18:12:45.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.186+0000 7f18eb7fe640 1 -- 172.21.15.46:0/2562405875 learned_addr learned my addr 172.21.15.46:0/2562405875 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614933923 0 0) 0x7f18ec0b2a30 con 0x7f18ec13f710 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:12:45.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.198+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d4003520 con 0x7f18ec13f710 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875379615 0 0) 0x7f18ec0b28a0 con 0x7f18ec1430c0 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ec0b2a30 con 0x7f18ec1430c0 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115384349 0 0) 0x7f18ec142520 con 0x7f18ec0a98f0 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.204+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ec0b28a0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.216+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228396120 0 0) 0x7f18d4003520 con 0x7f18ec13f710 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.216+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 >> v1:172.21.15.164:6789/0 conn(0x7f18ec1430c0 legacy=0x7f18ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.216+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 >> v1:172.21.15.46:6789/0 conn(0x7f18ec0a98f0 legacy=0x7f18ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.216+0000 7f18ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.216+0000 7f18ea7fc640 1 monclient: found mon.noname-b 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0b44c0 con 0x7f18ec13f710 2024-03-25T18:12:45.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f18e0001690 con 0x7f18ec13f710 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18ea7fc640 10 monclient: dump: 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:45.219 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:45.220 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18f203e640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:12:45.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.217+0000 7f18f203e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18e00017f0 con 0x7f18ec13f710 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 1 -- 172.21.15.46:0/2562405875 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f18e0001c00 con 0x7f18ec13f710 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18ea7fc640 10 monclient: dump: 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:45.230 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18f203e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:45.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.229+0000 7f18f203e640 4 set_mon_vals no callback set 2024-03-25T18:12:45.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 10 monclient: shutdown 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 1 -- 172.21.15.46:0/2562405875 >> v1:172.21.15.46:6790/0 conn(0x7f18ec13f710 legacy=0x7f18ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 1 -- 172.21.15.46:0/2562405875 shutdown_connections 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 1 -- 172.21.15.46:0/2562405875 >> 172.21.15.46:0/2562405875 conn(0x7f18ec0b7ea0 msgr2=0x7f18ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.230+0000 7f18f203e640 1 -- 172.21.15.46:0/2562405875 shutdown_connections 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.231+0000 7f18f203e640 1 -- 172.21.15.46:0/2562405875 wait complete. 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.231+0000 7f18f203e640 10 monclient: build_initial_monmap 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.231+0000 7f18f203e640 10 monclient: monmap: 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:45.233 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 librados: starting msgr at 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 librados: starting objecter 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 Processor -- start 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 -- start start 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 librados: setting wanted keys 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 1 librados: calling monclient init 2024-03-25T18:12:45.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.232+0000 7f18f203e640 10 monclient: init 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-b con 0x7f18ec0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-a con 0x7f18ec13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient(hunting): picked mon.noname-c con 0x7f18ec1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec0ab8b0 con 0x7f18ec13f710 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec0abb90 con 0x7f18ec0a98f0 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec137870 con 0x7f18ec1430c0 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:45.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.233+0000 7f18f203e640 10 monclient(hunting): authenticate will time out at 7474.495189s 2024-03-25T18:12:45.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.244+0000 7f18eb7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f18ec0a98f0 0x7f18ec1350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45216/0 (socket says 172.21.15.46:45216) 2024-03-25T18:12:45.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.244+0000 7f18eb7fe640 1 -- 172.21.15.46:0/345645333 learned_addr learned my addr 172.21.15.46:0/345645333 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:45.245 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.244+0000 7f18d3fff640 10 client.?.objecter ms_handle_connect 0x7f18ec0a98f0 2024-03-25T18:12:45.245 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.244+0000 7f18d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023030579 0 0) 0x7f18ec0abb90 con 0x7f18ec0a98f0 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:12:45.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.252+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d8004eb0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.253+0000 7f18d3fff640 10 client.?.objecter ms_handle_connect 0x7f18ec13f710 2024-03-25T18:12:45.254 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.253+0000 7f18d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.255+0000 7f18d3fff640 10 client.?.objecter ms_handle_connect 0x7f18ec1430c0 2024-03-25T18:12:45.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.255+0000 7f18d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463215245 0 0) 0x7f18ec0ab8b0 con 0x7f18ec13f710 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:12:45.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.265+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ec0abb90 con 0x7f18ec13f710 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974826697 0 0) 0x7f18d8004eb0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18ec0ab8b0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f18dc0018b0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 10 monclient(hunting): dump: 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.269 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.270 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.268+0000 7f18d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709314152 0 0) 0x7f18ec137870 con 0x7f18ec1430c0 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:12:45.272 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.271+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d8004eb0 con 0x7f18ec1430c0 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 78792131 0 0) 0x7f18ec0ab8b0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 >> v1:172.21.15.46:6790/0 conn(0x7f18ec1430c0 legacy=0x7f18ec1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 >> v1:172.21.15.46:6789/0 conn(0x7f18ec13f710 legacy=0x7f18ec1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 1 monclient: found mon.b 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec1389e0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 10 monclient: _renew_subs 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ec136d90 con 0x7f18ec0a98f0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 10 monclient: _renew_subs 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ec1384c0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.280+0000 7f18f203e640 1 librados: init done 2024-03-25T18:12:45.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.281+0000 7f18f203e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.281+0000 7f18f203e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.281+0000 7f18f203e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:45.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.281+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ec136f80 con 0x7f18ec0a98f0 2024-03-25T18:12:45.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18dc0047e0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:45.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f18dc004bb0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 10 monclient: dump: 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:45.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18e9ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:45.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.555+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f18dc004dd0 con 0x7f18ec0a98f0 2024-03-25T18:12:45.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.556+0000 7f18eaffd640 10 monclient: get_auth_request con 0x7f18d806ddf0 auth_method 0 2024-03-25T18:12:45.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.556+0000 7f18d3fff640 10 client.4405.objecter ms_handle_connect 0x7f18d806ddf0 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f18dc054820 con 0x7f18ec0a98f0 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 10 client.4405.objecter ms_dispatch 0x7f18ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18dc050330 con 0x7f18ec0a98f0 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.562+0000 7f18d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:45.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.787+0000 7f18f203e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.787+0000 7f18f203e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.787+0000 7f18f203e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:45.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.787+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18ec138b70 con 0x7f18ec0a98f0 2024-03-25T18:12:45.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.811+0000 7f18d3fff640 1 -- 172.21.15.46:0/345645333 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f18dc052260 con 0x7f18ec0a98f0 2024-03-25T18:12:45.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.811+0000 7f18d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.811+0000 7f18d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:45.812 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:45.812 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 10 librados: watch_flush enter 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 10 librados: watch_flush exit 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 >> v1:172.21.15.164:6800/1612084612 conn(0x7f18d806ddf0 legacy=0x7f18d80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 10 monclient: shutdown 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.813+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 >> v1:172.21.15.164:6789/0 conn(0x7f18ec0a98f0 legacy=0x7f18ec1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.814+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 shutdown_connections 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.814+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 >> 172.21.15.46:0/345645333 conn(0x7f18ec0b7ea0 msgr2=0x7f18ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.814+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 shutdown_connections 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.814+0000 7f18f203e640 1 -- 172.21.15.46:0/345645333 wait complete. 2024-03-25T18:12:45.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:45.814+0000 7f18f203e640 1 librados: shutdown 2024-03-25T18:12:45.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:52.632 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.746+0000 7f04ea1d0640 10 monclient: get_monmap_and_config 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.746+0000 7f04ea1d0640 10 monclient: build_initial_monmap 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.746+0000 7f04ea1d0640 10 monclient: monmap: 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:12:52.747047+0000 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:12:52.747047+0000 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:52.747 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:12:52.748 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:12:52.748 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.747+0000 7f04ea1d0640 1 Processor -- start 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.747+0000 7f04ea1d0640 1 -- start start 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.747+0000 7f04ea1d0640 10 monclient: init 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-c con 0x7f04e40a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-b con 0x7f04e413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-a con 0x7f04e41430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e4142520 con 0x7f04e41430c0 2024-03-25T18:12:52.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e40b28a0 con 0x7f04e40a98f0 2024-03-25T18:12:52.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e40b2a30 con 0x7f04e413f710 2024-03-25T18:12:52.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:52.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.748+0000 7f04ea1d0640 10 monclient(hunting): authenticate will time out at 7482.009843s 2024-03-25T18:12:52.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.758+0000 7f04e3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f04e41430c0 0x7f04e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41388/0 (socket says 172.21.15.46:41388) 2024-03-25T18:12:52.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.758+0000 7f04e3fff640 1 -- 172.21.15.46:0/2235624689 learned_addr learned my addr 172.21.15.46:0/2235624689 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966294989 0 0) 0x7f04e4142520 con 0x7f04e41430c0 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.774+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc003520 con 0x7f04e41430c0 2024-03-25T18:12:52.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173895356 0 0) 0x7f04e40b2a30 con 0x7f04e413f710 2024-03-25T18:12:52.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:12:52.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.775+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e4142520 con 0x7f04e413f710 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682967247 0 0) 0x7f04e40b28a0 con 0x7f04e40a98f0 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:12:52.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.780+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e40b2a30 con 0x7f04e40a98f0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306518245 0 0) 0x7f04cc003520 con 0x7f04e41430c0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 >> v1:172.21.15.46:6790/0 conn(0x7f04e413f710 legacy=0x7f04e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 >> v1:172.21.15.164:6789/0 conn(0x7f04e40a98f0 legacy=0x7f04e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 monclient: found mon.noname-a 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e40b44c0 con 0x7f04e41430c0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f04dc001a10 con 0x7f04e41430c0 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04e27fc640 10 monclient: dump: 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04ea1d0640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:12:52.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.794+0000 7f04ea1d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:52.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04dc001b70 con 0x7f04e41430c0 2024-03-25T18:12:52.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:52.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 1 -- 172.21.15.46:0/2235624689 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f04dc001f80 con 0x7f04e41430c0 2024-03-25T18:12:52.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04e27fc640 10 monclient: dump: 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04ea1d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:52.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.804+0000 7f04ea1d0640 4 set_mon_vals no callback set 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 10 monclient: shutdown 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2235624689 >> v1:172.21.15.46:6789/0 conn(0x7f04e41430c0 legacy=0x7f04e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2235624689 shutdown_connections 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2235624689 >> 172.21.15.46:0/2235624689 conn(0x7f04e40b7ea0 msgr2=0x7f04e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2235624689 shutdown_connections 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.805+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2235624689 wait complete. 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.806+0000 7f04ea1d0640 10 monclient: build_initial_monmap 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.806+0000 7f04ea1d0640 10 monclient: monmap: 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:52.808 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.806+0000 7f04ea1d0640 1 librados: starting msgr at 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.806+0000 7f04ea1d0640 1 librados: starting objecter 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.807+0000 7f04ea1d0640 1 Processor -- start 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.807+0000 7f04ea1d0640 1 -- start start 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.807+0000 7f04ea1d0640 1 librados: setting wanted keys 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.807+0000 7f04ea1d0640 1 librados: calling monclient init 2024-03-25T18:12:52.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.807+0000 7f04ea1d0640 10 monclient: init 2024-03-25T18:12:52.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:52.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:52.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-b con 0x7f04e40a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:12:52.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-a con 0x7f04e413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:12:52.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient(hunting): picked mon.noname-c con 0x7f04e41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:12:52.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e4139830 con 0x7f04e413f710 2024-03-25T18:12:52.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e41db7d0 con 0x7f04e40a98f0 2024-03-25T18:12:52.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e41db410 con 0x7f04e41430c0 2024-03-25T18:12:52.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:52.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.808+0000 7f04ea1d0640 10 monclient(hunting): authenticate will time out at 7482.069808s 2024-03-25T18:12:52.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.824+0000 7f04e3fff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f04e41430c0 0x7f04e41d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:44622/0 (socket says 172.21.15.46:44622) 2024-03-25T18:12:52.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.824+0000 7f04e3fff640 1 -- 172.21.15.46:0/2517280323 learned_addr learned my addr 172.21.15.46:0/2517280323 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:12:52.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.825+0000 7f04cbfff640 10 client.?.objecter ms_handle_connect 0x7f04e41430c0 2024-03-25T18:12:52.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.825+0000 7f04cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.832+0000 7f04cbfff640 10 client.?.objecter ms_handle_connect 0x7f04e413f710 2024-03-25T18:12:52.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.832+0000 7f04cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.837 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.836+0000 7f04cbfff640 10 client.?.objecter ms_handle_connect 0x7f04e40a98f0 2024-03-25T18:12:52.837 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.836+0000 7f04cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398868096 0 0) 0x7f04e41db410 con 0x7f04e41430c0 2024-03-25T18:12:52.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:12:52.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.842+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0004eb0 con 0x7f04e41430c0 2024-03-25T18:12:52.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141256882 0 0) 0x7f04e41db7d0 con 0x7f04e40a98f0 2024-03-25T18:12:52.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:12:52.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.844+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e41db410 con 0x7f04e40a98f0 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553142042 0 0) 0x7f04e4139830 con 0x7f04e413f710 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:12:52.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.847+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e41db7d0 con 0x7f04e413f710 2024-03-25T18:12:52.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.855+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 267473428 0 0) 0x7f04e41db410 con 0x7f04e40a98f0 2024-03-25T18:12:52.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.855+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e4139830 con 0x7f04e40a98f0 2024-03-25T18:12:52.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.856+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f04d4001870 con 0x7f04e40a98f0 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.856+0000 7f04cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.856+0000 7f04cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.856+0000 7f04cbfff640 10 monclient(hunting): dump: 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:52.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.856+0000 7f04cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:52.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2045798382 0 0) 0x7f04e4139830 con 0x7f04e40a98f0 2024-03-25T18:12:52.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 >> v1:172.21.15.46:6790/0 conn(0x7f04e41430c0 legacy=0x7f04e41d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 >> v1:172.21.15.46:6789/0 conn(0x7f04e413f710 legacy=0x7f04e40b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 1 monclient: found mon.b 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e41db5a0 con 0x7f04e40a98f0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 10 monclient: _finish_auth 0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 10 monclient: _renew_subs 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e40b1480 con 0x7f04e40a98f0 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 10 monclient: _renew_subs 2024-03-25T18:12:52.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e40b0b10 con 0x7f04e40a98f0 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.862+0000 7f04ea1d0640 1 librados: init done 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.863+0000 7f04ea1d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.863+0000 7f04ea1d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.863+0000 7f04ea1d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:52.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:52.863+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e41db960 con 0x7f04e40a98f0 2024-03-25T18:12:53.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.076+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d4003610 con 0x7f04e40a98f0 2024-03-25T18:12:53.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.076+0000 7f04cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:53.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.076+0000 7f04e1ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:53.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.139+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f04d4001d80 con 0x7f04e40a98f0 2024-03-25T18:12:53.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.139+0000 7f04cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.139+0000 7f04cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.139+0000 7f04cbfff640 10 monclient: dump: 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:12:53.140 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:12:53.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.140+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f04d4003ae0 con 0x7f04e40a98f0 2024-03-25T18:12:53.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.140+0000 7f04e2ffd640 10 monclient: get_auth_request con 0x7f04d006de20 auth_method 0 2024-03-25T18:12:53.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.141+0000 7f04cbfff640 10 client.4411.objecter ms_handle_connect 0x7f04d006de20 2024-03-25T18:12:53.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.149+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f04d4053600 con 0x7f04e40a98f0 2024-03-25T18:12:53.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.149+0000 7f04cbfff640 10 client.4411.objecter ms_dispatch 0x7f04e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:53.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.149+0000 7f04cbfff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:53.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.149+0000 7f04cbfff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:53.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.149+0000 7f04cbfff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:12:53.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.153+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04d404f110 con 0x7f04e40a98f0 2024-03-25T18:12:53.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.153+0000 7f04cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:53.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.153+0000 7f04cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.408+0000 7f04ea1d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.408+0000 7f04ea1d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.408+0000 7f04ea1d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:12:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.408+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e40627b0 con 0x7f04e40a98f0 2024-03-25T18:12:53.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.448+0000 7f04cbfff640 1 -- 172.21.15.46:0/2517280323 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f04d4051040 con 0x7f04e40a98f0 2024-03-25T18:12:53.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.448+0000 7f04cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.448+0000 7f04cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:53.450 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:12:53.450 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 10 librados: watch_flush enter 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 10 librados: watch_flush exit 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 >> v1:172.21.15.164:6800/1612084612 conn(0x7f04d006de20 legacy=0x7f04d00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 10 monclient: shutdown 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 >> v1:172.21.15.164:6789/0 conn(0x7f04e40a98f0 legacy=0x7f04e41dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 shutdown_connections 2024-03-25T18:12:53.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 >> 172.21.15.46:0/2517280323 conn(0x7f04e40b7ea0 msgr2=0x7f04e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:53.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 shutdown_connections 2024-03-25T18:12:53.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 -- 172.21.15.46:0/2517280323 wait complete. 2024-03-25T18:12:53.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:12:53.451+0000 7f04ea1d0640 1 librados: shutdown 2024-03-25T18:12:53.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:59.193 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:35.110 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.243 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:35.160 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.266 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:35.184 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.293 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:35.164 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.295 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:35.212 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.298 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:35.169 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.311 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:35.229 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.335 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:35.206 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:59.355 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:35.273 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:00.270 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:00.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.386+0000 7fda13eff640 10 monclient: get_monmap_and_config 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.386+0000 7fda13eff640 10 monclient: build_initial_monmap 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.386+0000 7fda13eff640 10 monclient: monmap: 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:00.387497+0000 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:00.387497+0000 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:00.388 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.387+0000 7fda13eff640 1 Processor -- start 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.387+0000 7fda13eff640 1 -- start start 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.387+0000 7fda13eff640 10 monclient: init 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.388+0000 7fda13eff640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.388+0000 7fda13eff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.388+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-a con 0x7fda0c06ea40 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.388+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-c con 0x7fda0c071800 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-b con 0x7fda0c071c40 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c06f1f0 con 0x7fda0c06ea40 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c070d00 con 0x7fda0c071800 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c0720b0 con 0x7fda0c071c40 2024-03-25T18:13:00.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:00.391 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.389+0000 7fda13eff640 10 monclient(hunting): authenticate will time out at 7489.650729s 2024-03-25T18:13:00.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.395+0000 7fda1249c640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fda0c071c40 0x7fda0c14a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:50204/0 (socket says 172.21.15.46:50204) 2024-03-25T18:13:00.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.395+0000 7fda1249c640 1 -- 172.21.15.46:0/3020491852 learned_addr learned my addr 172.21.15.46:0/3020491852 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:00.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423274449 0 0) 0x7fda0c0720b0 con 0x7fda0c071c40 2024-03-25T18:13:00.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:13:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.408+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f8003520 con 0x7fda0c071c40 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480994839 0 0) 0x7fda0c070d00 con 0x7fda0c071800 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:13:00.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.412+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c0720b0 con 0x7fda0c071800 2024-03-25T18:13:00.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375659823 0 0) 0x7fda0c06f1f0 con 0x7fda0c06ea40 2024-03-25T18:13:00.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:13:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.420+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c070d00 con 0x7fda0c06ea40 2024-03-25T18:13:00.426 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716577644 0 0) 0x7fda0c0720b0 con 0x7fda0c071800 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 >> v1:172.21.15.46:6790/0 conn(0x7fda0c071c40 legacy=0x7fda0c14a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 >> v1:172.21.15.46:6789/0 conn(0x7fda0c06ea40 legacy=0x7fda0c13ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: _finish_hunting 0 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 monclient: found mon.noname-c 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c072780 con 0x7fda0c071800 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: _finish_auth 0 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: _check_auth_tickets 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9fc001670 con 0x7fda0c071800 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:00.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda10c99640 10 monclient: dump: 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda13eff640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:13:00.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.425+0000 7fda13eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:00.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9fc0017d0 con 0x7fda0c071800 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 1 -- 172.21.15.46:0/3020491852 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9fc001be0 con 0x7fda0c071800 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda10c99640 10 monclient: dump: 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.434 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.435 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.435 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda13eff640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:00.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.433+0000 7fda13eff640 4 set_mon_vals no callback set 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 10 monclient: shutdown 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 1 -- 172.21.15.46:0/3020491852 >> v1:172.21.15.164:6789/0 conn(0x7fda0c071800 legacy=0x7fda0c147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 1 -- 172.21.15.46:0/3020491852 shutdown_connections 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 1 -- 172.21.15.46:0/3020491852 >> 172.21.15.46:0/3020491852 conn(0x7fda0c06b630 msgr2=0x7fda0c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 1 -- 172.21.15.46:0/3020491852 shutdown_connections 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.434+0000 7fda13eff640 1 -- 172.21.15.46:0/3020491852 wait complete. 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 10 monclient: build_initial_monmap 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 10 monclient: monmap: 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 librados: starting msgr at 2024-03-25T18:13:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 librados: starting objecter 2024-03-25T18:13:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 Processor -- start 2024-03-25T18:13:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 -- start start 2024-03-25T18:13:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 librados: setting wanted keys 2024-03-25T18:13:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 1 librados: calling monclient init 2024-03-25T18:13:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.435+0000 7fda13eff640 10 monclient: init 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-b con 0x7fda0c06ea40 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-c con 0x7fda0c071800 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient(hunting): picked mon.noname-a con 0x7fda0c071c40 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c11dc00 con 0x7fda0c071c40 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c11b7f0 con 0x7fda0c06ea40 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c139cc0 con 0x7fda0c071800 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.436+0000 7fda13eff640 10 monclient(hunting): authenticate will time out at 7489.698240s 2024-03-25T18:13:00.445 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.445+0000 7fda1149a640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fda0c071800 0x7fda0c1d7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:50228/0 (socket says 172.21.15.46:50228) 2024-03-25T18:13:00.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.445+0000 7fda1149a640 1 -- 172.21.15.46:0/3664195677 learned_addr learned my addr 172.21.15.46:0/3664195677 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:00.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.449+0000 7fda027fc640 10 client.?.objecter ms_handle_connect 0x7fda0c06ea40 2024-03-25T18:13:00.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.449+0000 7fda027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.449+0000 7fda027fc640 10 client.?.objecter ms_handle_connect 0x7fda0c071800 2024-03-25T18:13:00.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.449+0000 7fda027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224054032 0 0) 0x7fda0c139cc0 con 0x7fda0c071800 2024-03-25T18:13:00.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:13:00.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.460+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f0004eb0 con 0x7fda0c071800 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722911856 0 0) 0x7fda0c11b7f0 con 0x7fda0c06ea40 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c139cc0 con 0x7fda0c06ea40 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 client.?.objecter ms_handle_connect 0x7fda0c071c40 2024-03-25T18:13:00.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.467+0000 7fda027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744409820 0 0) 0x7fda0c11dc00 con 0x7fda0c071c40 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:13:00.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.492+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c11b7f0 con 0x7fda0c071c40 2024-03-25T18:13:00.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 59825 0 0) 0x7fda0c139cc0 con 0x7fda0c06ea40 2024-03-25T18:13:00.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda0c11dc00 con 0x7fda0c06ea40 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fda080018b0 con 0x7fda0c06ea40 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 10 monclient(hunting): dump: 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.495 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.494+0000 7fda027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:00.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029380371 0 0) 0x7fda0c11b7f0 con 0x7fda0c071c40 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda0c139cc0 con 0x7fda0c071c40 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fda040019e0 con 0x7fda0c071c40 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.502+0000 7fda027fc640 10 monclient(hunting): dump: 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.503 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321348810 0 0) 0x7fd9f0004eb0 con 0x7fda0c071800 2024-03-25T18:13:00.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda0c11b7f0 con 0x7fda0c071800 2024-03-25T18:13:00.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9fc001670 con 0x7fda0c071800 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.504+0000 7fda027fc640 10 monclient(hunting): dump: 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.505 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.511+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2216197857 0 0) 0x7fda0c11b7f0 con 0x7fda0c071800 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 >> v1:172.21.15.164:6789/0 conn(0x7fda0c06ea40 legacy=0x7fda0c1395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 >> v1:172.21.15.46:6789/0 conn(0x7fda0c071c40 legacy=0x7fda0c1d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 1 monclient: found mon.c 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c139e50 con 0x7fda0c071800 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:00.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 10 monclient: _renew_subs 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c13a0a0 con 0x7fda0c071800 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 10 monclient: _renew_subs 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c11bef0 con 0x7fda0c071800 2024-03-25T18:13:00.514 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.512+0000 7fda13eff640 1 librados: init done 2024-03-25T18:13:00.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.513+0000 7fda13eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:00.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.513+0000 7fda13eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:00.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.513+0000 7fda13eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:00.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.513+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c074d80 con 0x7fda0c071800 2024-03-25T18:13:00.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9fc0038a0 con 0x7fda0c071800 2024-03-25T18:13:00.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9fc003ad0 con 0x7fda0c071800 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda027fc640 10 monclient: dump: 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:00.845 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:00.846 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:00.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.843+0000 7fda03fff640 4 set_mon_vals no callback set 2024-03-25T18:13:00.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.844+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fd9fc003cf0 con 0x7fda0c071800 2024-03-25T18:13:00.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.845+0000 7fda11c9b640 10 monclient: get_auth_request con 0x7fd9f006e160 auth_method 0 2024-03-25T18:13:00.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.845+0000 7fda027fc640 10 client.4424.objecter ms_handle_connect 0x7fd9f006e160 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fd9fc0547c0 con 0x7fda0c071800 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 10 client.4424.objecter ms_dispatch 0x7fda0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9fc0502d0 con 0x7fda0c071800 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:00.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:00.851+0000 7fda027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:01.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.080+0000 7fda13eff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.080+0000 7fda13eff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.080+0000 7fda13eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:01.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.080+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0c0627b0 con 0x7fda0c071800 2024-03-25T18:13:01.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.133+0000 7fda027fc640 1 -- 172.21.15.46:0/3664195677 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fd9fc052200 con 0x7fda0c071800 2024-03-25T18:13:01.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.133+0000 7fda027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.133+0000 7fda027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:01.134 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:01.134 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 10 librados: watch_flush enter 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 10 librados: watch_flush exit 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 >> v1:172.21.15.164:6800/1612084612 conn(0x7fd9f006e160 legacy=0x7fd9f0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:01.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 10 monclient: shutdown 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 >> v1:172.21.15.46:6790/0 conn(0x7fda0c071800 legacy=0x7fda0c1d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 shutdown_connections 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 >> 172.21.15.46:0/3664195677 conn(0x7fda0c06b630 msgr2=0x7fda0c0b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 shutdown_connections 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 -- 172.21.15.46:0/3664195677 wait complete. 2024-03-25T18:13:01.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:01.135+0000 7fda13eff640 1 librados: shutdown 2024-03-25T18:13:01.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:07.954 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:08.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.073+0000 7f6b42a63640 10 monclient: get_monmap_and_config 2024-03-25T18:13:08.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.073+0000 7f6b42a63640 10 monclient: build_initial_monmap 2024-03-25T18:13:08.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.073+0000 7f6b42a63640 10 monclient: monmap: 2024-03-25T18:13:08.074 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:08.074 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:08.073785+0000 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:08.073785+0000 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 1 Processor -- start 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 1 -- start start 2024-03-25T18:13:08.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 10 monclient: init 2024-03-25T18:13:08.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.074+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c142520 con 0x7f6b3c1430c0 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0b28a0 con 0x7f6b3c13f710 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0b2a30 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:08.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.075+0000 7f6b42a63640 10 monclient(hunting): authenticate will time out at 7497.336706s 2024-03-25T18:13:08.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.077+0000 7f6b3bfff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f6b3c0a98f0 0x7f6b3c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:58318/0 (socket says 172.21.15.46:58318) 2024-03-25T18:13:08.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.077+0000 7f6b3bfff640 1 -- 172.21.15.46:0/3198747647 learned_addr learned my addr 172.21.15.46:0/3198747647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:08.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859265456 0 0) 0x7f6b3c0b2a30 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:13:08.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.101+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b28003520 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783964924 0 0) 0x7f6b3c142520 con 0x7f6b3c1430c0 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:13:08.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.103+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c0b2a30 con 0x7f6b3c1430c0 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541131169 0 0) 0x7f6b3c0b28a0 con 0x7f6b3c13f710 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:13:08.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.111+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c142520 con 0x7f6b3c13f710 2024-03-25T18:13:08.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3143347938 0 0) 0x7f6b28003520 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 >> v1:172.21.15.164:6789/0 conn(0x7f6b3c13f710 legacy=0x7f6b3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 >> v1:172.21.15.46:6789/0 conn(0x7f6b3c1430c0 legacy=0x7f6b3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 monclient: found mon.noname-b 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0b44c0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: _finish_auth 0 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b2c001a10 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b3affd640 10 monclient: dump: 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.117 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b42a63640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:13:08.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.115+0000 7f6b42a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:08.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b2c001b70 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:08.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 1 -- 172.21.15.46:0/3198747647 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b2c001f80 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 4 set_mon_vals no callback set 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b3affd640 10 monclient: dump: 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.132 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 10 monclient: shutdown 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 1 -- 172.21.15.46:0/3198747647 >> v1:172.21.15.46:6790/0 conn(0x7f6b3c0a98f0 legacy=0x7f6b3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 1 -- 172.21.15.46:0/3198747647 shutdown_connections 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 1 -- 172.21.15.46:0/3198747647 >> 172.21.15.46:0/3198747647 conn(0x7f6b3c0b7ea0 msgr2=0x7f6b3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 1 -- 172.21.15.46:0/3198747647 shutdown_connections 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.131+0000 7f6b42a63640 1 -- 172.21.15.46:0/3198747647 wait complete. 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.132+0000 7f6b42a63640 10 monclient: build_initial_monmap 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.132+0000 7f6b42a63640 10 monclient: monmap: 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:08.135 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 librados: starting msgr at 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 librados: starting objecter 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 Processor -- start 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 -- start start 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 librados: setting wanted keys 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 1 librados: calling monclient init 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient: init 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:08.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.133+0000 7f6b42a63640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.134+0000 7f6b42a63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c13be70 con 0x7f6b3c13f710 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.134+0000 7f6b42a63640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c131660 con 0x7f6b3c1430c0 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.134+0000 7f6b42a63640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c1317f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.134+0000 7f6b42a63640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:08.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.134+0000 7f6b42a63640 10 monclient(hunting): authenticate will time out at 7497.395631s 2024-03-25T18:13:08.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.141+0000 7f6b3bfff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f6b3c0a98f0 0x7f6b3c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:58348/0 (socket says 172.21.15.46:58348) 2024-03-25T18:13:08.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.141+0000 7f6b3bfff640 1 -- 172.21.15.46:0/3162309159 learned_addr learned my addr 172.21.15.46:0/3162309159 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:08.145 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.145+0000 7f6b38ff9640 10 client.?.objecter ms_handle_connect 0x7f6b3c1430c0 2024-03-25T18:13:08.145 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.145+0000 7f6b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.146+0000 7f6b38ff9640 10 client.?.objecter ms_handle_connect 0x7f6b3c0a98f0 2024-03-25T18:13:08.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.146+0000 7f6b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.158 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.158+0000 7f6b38ff9640 10 client.?.objecter ms_handle_connect 0x7f6b3c13f710 2024-03-25T18:13:08.158 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.158+0000 7f6b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.160+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769135588 0 0) 0x7f6b3c1317f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.160+0000 7f6b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.160+0000 7f6b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.160+0000 7f6b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.160+0000 7f6b38ff9640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:13:08.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.161+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1c004eb0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152567677 0 0) 0x7f6b3c131660 con 0x7f6b3c1430c0 2024-03-25T18:13:08.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:13:08.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.163+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c1317f0 con 0x7f6b3c1430c0 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354556993 0 0) 0x7f6b3c13be70 con 0x7f6b3c13f710 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:13:08.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.178+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c131660 con 0x7f6b3c13f710 2024-03-25T18:13:08.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241476439 0 0) 0x7f6b3c1317f0 con 0x7f6b3c1430c0 2024-03-25T18:13:08.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b3c13be70 con 0x7f6b3c1430c0 2024-03-25T18:13:08.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b340017d0 con 0x7f6b3c1430c0 2024-03-25T18:13:08.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.180+0000 7f6b38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972887778 0 0) 0x7f6b1c004eb0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b3c1317f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b2c001f40 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.182+0000 7f6b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:08.183 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.184 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2513496770 0 0) 0x7f6b3c131660 con 0x7f6b3c13f710 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b1c004eb0 con 0x7f6b3c13f710 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b24001830 con 0x7f6b3c13f710 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:08.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.194+0000 7f6b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.196 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.197 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.197 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.197 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.197 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1944482676 0 0) 0x7f6b3c1317f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 >> v1:172.21.15.164:6789/0 conn(0x7f6b3c1430c0 legacy=0x7f6b3c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 >> v1:172.21.15.46:6789/0 conn(0x7f6b3c13f710 legacy=0x7f6b3c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:08.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 1 monclient: found mon.c 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c131980 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 10 monclient: _renew_subs 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c135c60 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 10 monclient: _renew_subs 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c0b7770 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.203+0000 7f6b42a63640 1 librados: init done 2024-03-25T18:13:08.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.204+0000 7f6b42a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.204+0000 7f6b42a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.204+0000 7f6b42a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.204+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c074cc0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b2c004e10 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6b2c005120 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b38ff9640 10 monclient: dump: 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:08.587 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.586+0000 7f6b3a7fc640 4 set_mon_vals no callback set 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.587+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f6b2c01b0f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.588 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.587+0000 7f6b3b7fe640 10 monclient: get_auth_request con 0x7f6b1c06e1f0 auth_method 0 2024-03-25T18:13:08.589 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.588+0000 7f6b38ff9640 10 client.4430.objecter ms_handle_connect 0x7f6b1c06e1f0 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f6b2c054cb0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 10 client.4430.objecter ms_dispatch 0x7f6b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b2c0507c0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.594+0000 7f6b38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:08.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.807+0000 7f6b42a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:08.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.807+0000 7f6b42a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:08.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.807+0000 7f6b42a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:08.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.807+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b3c1371d0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.838+0000 7f6b38ff9640 1 -- 172.21.15.46:0/3162309159 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f6b2c0526f0 con 0x7f6b3c0a98f0 2024-03-25T18:13:08.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.838+0000 7f6b38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:08.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.838+0000 7f6b38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:08.840 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:08.840 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 10 librados: watch_flush enter 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 10 librados: watch_flush exit 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 10 client.4430.objecter successfully canceled tick 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 >> v1:172.21.15.164:6800/1612084612 conn(0x7f6b1c06e1f0 legacy=0x7f6b1c0706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 10 monclient: shutdown 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 >> v1:172.21.15.46:6790/0 conn(0x7f6b3c0a98f0 legacy=0x7f6b3c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 shutdown_connections 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 >> 172.21.15.46:0/3162309159 conn(0x7f6b3c0b7ea0 msgr2=0x7f6b3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 shutdown_connections 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 -- 172.21.15.46:0/3162309159 wait complete. 2024-03-25T18:13:08.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:08.841+0000 7f6b42a63640 1 librados: shutdown 2024-03-25T18:13:08.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:15.661 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.778+0000 7f323fcc8640 10 monclient: get_monmap_and_config 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.778+0000 7f323fcc8640 10 monclient: build_initial_monmap 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.778+0000 7f323fcc8640 10 monclient: monmap: 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:15.779081+0000 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:15.779081+0000 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:15.779 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:15.780 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:15.780 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 1 Processor -- start 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 1 -- start start 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient: init 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-b con 0x7f32380705b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-a con 0x7f32381402d0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:15.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-c con 0x7f3238149ae0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:15.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380718c0 con 0x7f32381402d0 2024-03-25T18:13:15.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.779+0000 7f323fcc8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380714c0 con 0x7f3238149ae0 2024-03-25T18:13:15.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.780+0000 7f323fcc8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3238072b50 con 0x7f32380705b0 2024-03-25T18:13:15.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.780+0000 7f323fcc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:15.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.780+0000 7f323fcc8640 10 monclient(hunting): authenticate will time out at 7505.041569s 2024-03-25T18:13:15.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.788+0000 7f323da64640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f32380705b0 0x7f323813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:40216/0 (socket says 172.21.15.46:40216) 2024-03-25T18:13:15.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.788+0000 7f323da64640 1 -- 172.21.15.46:0/3851460525 learned_addr learned my addr 172.21.15.46:0/3851460525 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:15.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.799+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346920507 0 0) 0x7f3238072b50 con 0x7f32380705b0 2024-03-25T18:13:15.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.800+0000 7f323ca62640 10 monclient(hunting): _negotiate 2024-03-25T18:13:15.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.800+0000 7f323ca62640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:15.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.800+0000 7f323ca62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:15.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.800+0000 7f323ca62640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:13:15.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.800+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3224003520 con 0x7f32380705b0 2024-03-25T18:13:15.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405205212 0 0) 0x7f32380718c0 con 0x7f32381402d0 2024-03-25T18:13:15.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 10 monclient(hunting): _negotiate 2024-03-25T18:13:15.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:15.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:15.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:13:15.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.806+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3238072b50 con 0x7f32381402d0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1120986705 0 0) 0x7f3224003520 con 0x7f32380705b0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 >> v1:172.21.15.164:6789/0 conn(0x7f3238149ae0 legacy=0x7f323814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 >> v1:172.21.15.46:6789/0 conn(0x7f32381402d0 legacy=0x7f3238140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: _finish_hunting 0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 monclient: found mon.noname-b 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323813dd10 con 0x7f32380705b0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: _finish_auth 0 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: _check_auth_tickets 2024-03-25T18:13:15.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f3234001690 con 0x7f32380705b0 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323fcc8640 5 monclient: authenticate success, global_id 4433 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323fcc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323ca62640 10 monclient: dump: 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:15.811 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:15.812 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.808+0000 7f323fcc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:15.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32340017f0 con 0x7f32380705b0 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 1 -- 172.21.15.46:0/3851460525 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f3234001c00 con 0x7f32380705b0 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323fcc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323fcc8640 4 set_mon_vals no callback set 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.817+0000 7f323ca62640 10 monclient: dump: 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.818 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:15.819 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:15.819 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:15.819 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:15.819 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:15.819 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 10 monclient: shutdown 2024-03-25T18:13:15.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:15.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 1 -- 172.21.15.46:0/3851460525 >> v1:172.21.15.46:6790/0 conn(0x7f32380705b0 legacy=0x7f323813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 1 -- 172.21.15.46:0/3851460525 shutdown_connections 2024-03-25T18:13:15.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 1 -- 172.21.15.46:0/3851460525 >> 172.21.15.46:0/3851460525 conn(0x7f32380b5d10 msgr2=0x7f32380b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 1 -- 172.21.15.46:0/3851460525 shutdown_connections 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.818+0000 7f323fcc8640 1 -- 172.21.15.46:0/3851460525 wait complete. 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 10 monclient: build_initial_monmap 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 10 monclient: monmap: 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:15.821 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 librados: starting msgr at 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 librados: starting objecter 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 Processor -- start 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 -- start start 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 librados: setting wanted keys 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 1 librados: calling monclient init 2024-03-25T18:13:15.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.819+0000 7f323fcc8640 10 monclient: init 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-b con 0x7f32380705b0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-c con 0x7f32381402d0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient(hunting): picked mon.noname-a con 0x7f3238149ae0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3238141f90 con 0x7f3238149ae0 2024-03-25T18:13:15.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323806d1c0 con 0x7f32380705b0 2024-03-25T18:13:15.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323806ce50 con 0x7f32381402d0 2024-03-25T18:13:15.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:15.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.820+0000 7f323fcc8640 10 monclient(hunting): authenticate will time out at 7505.082532s 2024-03-25T18:13:15.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.837+0000 7f323e265640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3238149ae0 0x7f3238134db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47516/0 (socket says 172.21.15.46:47516) 2024-03-25T18:13:15.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.837+0000 7f323e265640 1 -- 172.21.15.46:0/1818516984 learned_addr learned my addr 172.21.15.46:0/1818516984 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:15.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.839+0000 7f322e7fc640 10 client.?.objecter ms_handle_connect 0x7f32380705b0 2024-03-25T18:13:15.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.839+0000 7f322e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:15.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.840+0000 7f322e7fc640 10 client.?.objecter ms_handle_connect 0x7f3238149ae0 2024-03-25T18:13:15.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.840+0000 7f322e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:15.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716735574 0 0) 0x7f323806d1c0 con 0x7f32380705b0 2024-03-25T18:13:15.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:15.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:15.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:15.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:13:15.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.849+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c004eb0 con 0x7f32380705b0 2024-03-25T18:13:15.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.850+0000 7f322e7fc640 10 client.?.objecter ms_handle_connect 0x7f32381402d0 2024-03-25T18:13:15.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.850+0000 7f322e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435768854 0 0) 0x7f3238141f90 con 0x7f3238149ae0 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:13:15.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.854+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f323806d1c0 con 0x7f3238149ae0 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1879799691 0 0) 0x7f321c004eb0 con 0x7f32380705b0 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3238141f90 con 0x7f32380705b0 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f3234001b00 con 0x7f32380705b0 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:15.864 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:15.865 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:15.865 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:15.865 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.863+0000 7f322e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778856375 0 0) 0x7f323806d1c0 con 0x7f3238149ae0 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f321c004eb0 con 0x7f3238149ae0 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f3230001bb0 con 0x7f3238149ae0 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:15.870 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564136555 0 0) 0x7f323806ce50 con 0x7f32381402d0 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:13:15.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.869+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f323806d1c0 con 0x7f32381402d0 2024-03-25T18:13:15.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2797433794 0 0) 0x7f321c004eb0 con 0x7f3238149ae0 2024-03-25T18:13:15.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 >> v1:172.21.15.46:6790/0 conn(0x7f32381402d0 legacy=0x7f32381daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 >> v1:172.21.15.164:6789/0 conn(0x7f32380705b0 legacy=0x7f323806c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 1 monclient: found mon.a 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323806d6e0 con 0x7f3238149ae0 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f322e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f323fcc8640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f323fcc8640 10 monclient: _renew_subs 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f323fcc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:13:15.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.888+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32381db120 con 0x7f3238149ae0 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.889+0000 7f323fcc8640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.889+0000 7f323fcc8640 10 monclient: _renew_subs 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.889+0000 7f323fcc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.889+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32381490c0 con 0x7f3238149ae0 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.889+0000 7f323fcc8640 1 librados: init done 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.890+0000 7f323fcc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.890+0000 7f323fcc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:15.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.890+0000 7f323fcc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:13:15.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:15.890+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323806d350 con 0x7f3238149ae0 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32300020a0 con 0x7f3238149ae0 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f3230003b30 con 0x7f3238149ae0 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322e7fc640 10 monclient: dump: 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:16.214 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:16.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.213+0000 7f322ffff640 4 set_mon_vals no callback set 2024-03-25T18:13:16.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.214+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f3230003d50 con 0x7f3238149ae0 2024-03-25T18:13:16.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.214+0000 7f323da64640 10 monclient: get_auth_request con 0x7f321c06deb0 auth_method 0 2024-03-25T18:13:16.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.215+0000 7f322e7fc640 10 client.4428.objecter ms_handle_connect 0x7f321c06deb0 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f3230054a60 con 0x7f3238149ae0 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 10 client.4428.objecter ms_dispatch 0x7f3238000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:13:16.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32300503a0 con 0x7f3238149ae0 2024-03-25T18:13:16.221 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:16.221 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.219+0000 7f322e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:16.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.464+0000 7f323fcc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.464+0000 7f323fcc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.464+0000 7f323fcc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:13:16.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.464+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f323806cfe0 con 0x7f3238149ae0 2024-03-25T18:13:16.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.493+0000 7f322e7fc640 1 -- 172.21.15.46:0/1818516984 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f32300524a0 con 0x7f3238149ae0 2024-03-25T18:13:16.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.494+0000 7f322e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.494+0000 7f322e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:16.494 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:16.495 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 10 librados: watch_flush enter 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 10 librados: watch_flush exit 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 >> v1:172.21.15.164:6800/1612084612 conn(0x7f321c06deb0 legacy=0x7f321c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 10 monclient: shutdown 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 >> v1:172.21.15.46:6789/0 conn(0x7f3238149ae0 legacy=0x7f3238134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 shutdown_connections 2024-03-25T18:13:16.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 >> 172.21.15.46:0/1818516984 conn(0x7f32380b5d10 msgr2=0x7f32380b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:16.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 shutdown_connections 2024-03-25T18:13:16.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 -- 172.21.15.46:0/1818516984 wait complete. 2024-03-25T18:13:16.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:16.496+0000 7f323fcc8640 1 librados: shutdown 2024-03-25T18:13:16.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:19.309 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:55.227 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.319 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:55.237 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.356 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:04:55.274 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.375 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:55.292 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.381 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:55.252 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.382 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:55.299 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.390 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:55.261 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.414 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:04:55.332 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:19.420 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:04:55.292 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:23.313 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:23.432 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.430+0000 7ff4ffa8d640 10 monclient: get_monmap_and_config 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.430+0000 7ff4ffa8d640 10 monclient: build_initial_monmap 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.430+0000 7ff4ffa8d640 10 monclient: monmap: 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:23.432226+0000 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:23.432226+0000 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.431+0000 7ff4ffa8d640 1 Processor -- start 2024-03-25T18:13:23.433 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.431+0000 7ff4ffa8d640 1 -- start start 2024-03-25T18:13:23.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.431+0000 7ff4ffa8d640 10 monclient: init 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f8142520 con 0x7ff4f80a98f0 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80b28a0 con 0x7ff4f813f710 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80b2a30 con 0x7ff4f81430c0 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:23.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.432+0000 7ff4ffa8d640 10 monclient(hunting): authenticate will time out at 7512.694692s 2024-03-25T18:13:23.444 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.442+0000 7ff4fd028640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff4f813f710 0x7ff4f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47102/0 (socket says 172.21.15.46:47102) 2024-03-25T18:13:23.444 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.442+0000 7ff4fd028640 1 -- 172.21.15.46:0/3065909968 learned_addr learned my addr 172.21.15.46:0/3065909968 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966710424 0 0) 0x7ff4f8142520 con 0x7ff4f80a98f0 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:13:23.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.463+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e4003520 con 0x7ff4f80a98f0 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372070344 0 0) 0x7ff4f80b28a0 con 0x7ff4f813f710 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:13:23.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.466+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f8142520 con 0x7ff4f813f710 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531463145 0 0) 0x7ff4e4003520 con 0x7ff4f80a98f0 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 >> v1:172.21.15.46:6790/0 conn(0x7ff4f81430c0 legacy=0x7ff4f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 >> v1:172.21.15.164:6789/0 conn(0x7ff4f813f710 legacy=0x7ff4f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: _finish_hunting 0 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 monclient: found mon.noname-a 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:23.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f80b44c0 con 0x7ff4f80a98f0 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: _finish_auth 0 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: _check_auth_tickets 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4f4001690 con 0x7ff4f80a98f0 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4ffa8d640 5 monclient: authenticate success, global_id 4431 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4ffa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4fc827640 10 monclient: dump: 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.475 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.472+0000 7ff4ffa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f40017f0 con 0x7ff4f80a98f0 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 1 -- 172.21.15.46:0/3065909968 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4f4001c00 con 0x7ff4f80a98f0 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4ffa8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4ffa8d640 4 set_mon_vals no callback set 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.494+0000 7ff4fc827640 10 monclient: dump: 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.496 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.497 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:23.497 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:23.497 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:23.497 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:23.497 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 10 monclient: shutdown 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/3065909968 >> v1:172.21.15.46:6789/0 conn(0x7ff4f80a98f0 legacy=0x7ff4f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/3065909968 shutdown_connections 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/3065909968 >> 172.21.15.46:0/3065909968 conn(0x7ff4f80b7ea0 msgr2=0x7ff4f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/3065909968 shutdown_connections 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.495+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/3065909968 wait complete. 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 10 monclient: build_initial_monmap 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 10 monclient: monmap: 2024-03-25T18:13:23.499 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 librados: starting msgr at 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 librados: starting objecter 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 Processor -- start 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 -- start start 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 librados: setting wanted keys 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 1 librados: calling monclient init 2024-03-25T18:13:23.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.496+0000 7ff4ffa8d640 10 monclient: init 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80b7ce0 con 0x7ff4f80a98f0 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f81319b0 con 0x7ff4f813f710 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80b7800 con 0x7ff4f81430c0 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:23.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.497+0000 7ff4ffa8d640 10 monclient(hunting): authenticate will time out at 7512.760052s 2024-03-25T18:13:23.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.504+0000 7ff4fd028640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff4f813f710 0x7ff4f8130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47140/0 (socket says 172.21.15.46:47140) 2024-03-25T18:13:23.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.505+0000 7ff4fd028640 1 -- 172.21.15.46:0/267018437 learned_addr learned my addr 172.21.15.46:0/267018437 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:23.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.505+0000 7ff4ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff4f813f710 2024-03-25T18:13:23.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.505+0000 7ff4ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.512 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578858459 0 0) 0x7ff4f81319b0 con 0x7ff4f813f710 2024-03-25T18:13:23.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:13:23.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.511+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc004eb0 con 0x7ff4f813f710 2024-03-25T18:13:23.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.520+0000 7ff4ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff4f81430c0 2024-03-25T18:13:23.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.520+0000 7ff4ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.527+0000 7ff4ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff4f80a98f0 2024-03-25T18:13:23.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.527+0000 7ff4ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 890716840 0 0) 0x7ff4dc004eb0 con 0x7ff4f813f710 2024-03-25T18:13:23.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f81319b0 con 0x7ff4f813f710 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e80019b0 con 0x7ff4f813f710 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.530+0000 7ff4ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974887711 0 0) 0x7ff4f80b7800 con 0x7ff4f81430c0 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.537+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc004eb0 con 0x7ff4f81430c0 2024-03-25T18:13:23.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2127231316 0 0) 0x7ff4f81319b0 con 0x7ff4f813f710 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 >> v1:172.21.15.46:6790/0 conn(0x7ff4f81430c0 legacy=0x7ff4f81312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 >> v1:172.21.15.46:6789/0 conn(0x7ff4f80a98f0 legacy=0x7ff4f81dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 1 monclient: found mon.b 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8136f50 con 0x7ff4f813f710 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 10 monclient: _renew_subs 2024-03-25T18:13:23.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f81350f0 con 0x7ff4f813f710 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 10 monclient: _renew_subs 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f8135c60 con 0x7ff4f813f710 2024-03-25T18:13:23.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.538+0000 7ff4ffa8d640 1 librados: init done 2024-03-25T18:13:23.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.539+0000 7ff4ffa8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.539+0000 7ff4ffa8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.539+0000 7ff4ffa8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:23.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.539+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f80627b0 con 0x7ff4f813f710 2024-03-25T18:13:23.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4e80047e0 con 0x7ff4f813f710 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e8004b70 con 0x7ff4f813f710 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 10 monclient: dump: 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:23.819 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:23.820 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:23.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4effff640 4 set_mon_vals no callback set 2024-03-25T18:13:23.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.817+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7ff4e8004d90 con 0x7ff4f813f710 2024-03-25T18:13:23.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.818+0000 7ff4fd829640 10 monclient: get_auth_request con 0x7ff4dc06dd90 auth_method 0 2024-03-25T18:13:23.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.818+0000 7ff4ee7fc640 10 client.4435.objecter ms_handle_connect 0x7ff4dc06dd90 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7ff4e8054850 con 0x7ff4f813f710 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 10 client.4435.objecter ms_dispatch 0x7ff4f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:13:23.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4e8050360 con 0x7ff4f813f710 2024-03-25T18:13:23.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:23.822+0000 7ff4ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:24.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.044+0000 7ff4ffa8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.044+0000 7ff4ffa8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.044+0000 7ff4ffa8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:24.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.044+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f813e270 con 0x7ff4f813f710 2024-03-25T18:13:24.073 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.071+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/267018437 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7ff4e8052290 con 0x7ff4f813f710 2024-03-25T18:13:24.073 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.071+0000 7ff4ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.073 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.071+0000 7ff4ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:24.073 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:24.073 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.073+0000 7ff4ffa8d640 10 librados: watch_flush enter 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 10 librados: watch_flush exit 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 >> v1:172.21.15.164:6800/1612084612 conn(0x7ff4dc06dd90 legacy=0x7ff4dc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 10 monclient: shutdown 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 >> v1:172.21.15.164:6789/0 conn(0x7ff4f813f710 legacy=0x7ff4f8130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:24.076 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 shutdown_connections 2024-03-25T18:13:24.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 >> 172.21.15.46:0/267018437 conn(0x7ff4f80b7ea0 msgr2=0x7ff4f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:24.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 shutdown_connections 2024-03-25T18:13:24.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 -- 172.21.15.46:0/267018437 wait complete. 2024-03-25T18:13:24.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:24.074+0000 7ff4ffa8d640 1 librados: shutdown 2024-03-25T18:13:24.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:30.895 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:31.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.009+0000 7f137fee4640 10 monclient: get_monmap_and_config 2024-03-25T18:13:31.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.009+0000 7f137fee4640 10 monclient: build_initial_monmap 2024-03-25T18:13:31.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.009+0000 7f137fee4640 10 monclient: monmap: 2024-03-25T18:13:31.011 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:31.011 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:31.011552+0000 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:31.011552+0000 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:31.012 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.010+0000 7f137fee4640 1 Processor -- start 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.010+0000 7f137fee4640 1 -- start start 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.010+0000 7f137fee4640 10 monclient: init 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-a con 0x7f13780b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-b con 0x7f13780b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-c con 0x7f137814be90 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13780af940 con 0x7f13780b4d60 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13780b0a70 con 0x7f137814be90 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13780b0d50 con 0x7f13780b1730 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:31.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.011+0000 7f137fee4640 10 monclient(hunting): authenticate will time out at 7520.274093s 2024-03-25T18:13:31.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.027+0000 7f137e481640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f137814be90 0x7f137814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:34704/0 (socket says 172.21.15.46:34704) 2024-03-25T18:13:31.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.027+0000 7f137e481640 1 -- 172.21.15.46:0/1531722602 learned_addr learned my addr 172.21.15.46:0/1531722602 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123955111 0 0) 0x7f13780b0a70 con 0x7f137814be90 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:13:31.042 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.041+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1364003520 con 0x7f137814be90 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974731287 0 0) 0x7f13780b0d50 con 0x7f13780b1730 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:13:31.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.043+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13780b0a70 con 0x7f13780b1730 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151672261 0 0) 0x7f13780af940 con 0x7f13780b4d60 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:13:31.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.052+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13780b0d50 con 0x7f13780b4d60 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326345010 0 0) 0x7f13780b0a70 con 0x7f13780b1730 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 >> v1:172.21.15.164:6789/0 conn(0x7f137814be90 legacy=0x7f137814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 >> v1:172.21.15.46:6789/0 conn(0x7f13780b4d60 legacy=0x7f137813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: _finish_hunting 0 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 monclient: found mon.noname-b 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13780b1030 con 0x7f13780b1730 2024-03-25T18:13:31.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: _finish_auth 0 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: _check_auth_tickets 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1368001670 con 0x7f13780b1730 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137fee4640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137fee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137cc7e640 10 monclient: dump: 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.070 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.067+0000 7f137fee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13680017d0 con 0x7f13780b1730 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 1 -- 172.21.15.46:0/1531722602 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1368001be0 con 0x7f13780b1730 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137fee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137fee4640 4 set_mon_vals no callback set 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.104+0000 7f137cc7e640 10 monclient: dump: 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.106 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.107 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 10 monclient: shutdown 2024-03-25T18:13:31.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:31.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 1 -- 172.21.15.46:0/1531722602 >> v1:172.21.15.46:6790/0 conn(0x7f13780b1730 legacy=0x7f13780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 1 -- 172.21.15.46:0/1531722602 shutdown_connections 2024-03-25T18:13:31.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 1 -- 172.21.15.46:0/1531722602 >> 172.21.15.46:0/1531722602 conn(0x7f13780617a0 msgr2=0x7f1378061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 1 -- 172.21.15.46:0/1531722602 shutdown_connections 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.105+0000 7f137fee4640 1 -- 172.21.15.46:0/1531722602 wait complete. 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 10 monclient: build_initial_monmap 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 10 monclient: monmap: 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 librados: starting msgr at 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 librados: starting objecter 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 Processor -- start 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 -- start start 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 librados: setting wanted keys 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 1 librados: calling monclient init 2024-03-25T18:13:31.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.106+0000 7f137fee4640 10 monclient: init 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.107+0000 7f137fee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.107+0000 7f137fee4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.107+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-a con 0x7f13780b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.107+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-b con 0x7f13780b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.107+0000 7f137fee4640 10 monclient(hunting): picked mon.noname-c con 0x7f137814be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.108+0000 7f137fee4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13781dad30 con 0x7f13780b1730 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.108+0000 7f137fee4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1378143900 con 0x7f13780b4d60 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.108+0000 7f137fee4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1378140380 con 0x7f137814be90 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.108+0000 7f137fee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:31.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.108+0000 7f137fee4640 10 monclient(hunting): authenticate will time out at 7520.370646s 2024-03-25T18:13:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.118+0000 7f137e481640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f137814be90 0x7f13781db1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:42374/0 (socket says 172.21.15.46:42374) 2024-03-25T18:13:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.119+0000 7f137e481640 1 -- 172.21.15.46:0/901079929 learned_addr learned my addr 172.21.15.46:0/901079929 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:31.129 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.128+0000 7f136e7fc640 10 client.?.objecter ms_handle_connect 0x7f13780b1730 2024-03-25T18:13:31.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.128+0000 7f136e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.132+0000 7f136e7fc640 10 client.?.objecter ms_handle_connect 0x7f13780b4d60 2024-03-25T18:13:31.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.132+0000 7f136e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.134+0000 7f136e7fc640 10 client.?.objecter ms_handle_connect 0x7f137814be90 2024-03-25T18:13:31.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.134+0000 7f136e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528451490 0 0) 0x7f1378143900 con 0x7f13780b4d60 2024-03-25T18:13:31.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.146+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f135c004eb0 con 0x7f13780b4d60 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296545860 0 0) 0x7f13781dad30 con 0x7f13780b1730 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:13:31.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.147+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1378143900 con 0x7f13780b1730 2024-03-25T18:13:31.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291341335 0 0) 0x7f1378140380 con 0x7f137814be90 2024-03-25T18:13:31.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:13:31.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.151+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13781dad30 con 0x7f137814be90 2024-03-25T18:13:31.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2152724934 0 0) 0x7f135c004eb0 con 0x7f13780b4d60 2024-03-25T18:13:31.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1378140380 con 0x7f13780b4d60 2024-03-25T18:13:31.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1368001ba0 con 0x7f13780b4d60 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652326599 0 0) 0x7f1378143900 con 0x7f13780b1730 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f135c004eb0 con 0x7f13780b1730 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1374001870 con 0x7f13780b1730 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.160+0000 7f136e7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.163 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.164 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866849565 0 0) 0x7f13781dad30 con 0x7f137814be90 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1378143900 con 0x7f137814be90 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1370001ad0 con 0x7f137814be90 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:31.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.167+0000 7f136e7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.170 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1119023901 0 0) 0x7f1378140380 con 0x7f13780b4d60 2024-03-25T18:13:31.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 >> v1:172.21.15.46:6790/0 conn(0x7f137814be90 legacy=0x7f13781db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 >> v1:172.21.15.46:6789/0 conn(0x7f13780b1730 legacy=0x7f1378141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 1 monclient: found mon.b 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378141d50 con 0x7f13780b4d60 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f136e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f137fee4640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f137fee4640 10 monclient: _renew_subs 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f137fee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.174+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13781408a0 con 0x7f13780b4d60 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 10 monclient: _renew_subs 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13781336f0 con 0x7f13780b4d60 2024-03-25T18:13:31.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 1 librados: init done 2024-03-25T18:13:31.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.175+0000 7f137fee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.176+0000 7f137fee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.176+0000 7f137fee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:31.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.176+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1378140ba0 con 0x7f13780b4d60 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13680038a0 con 0x7f13780b4d60 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1368003a10 con 0x7f13780b4d60 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 10 monclient: dump: 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:31.478 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:31.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136ffff640 4 set_mon_vals no callback set 2024-03-25T18:13:31.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.476+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f13680199e0 con 0x7f13780b4d60 2024-03-25T18:13:31.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.477+0000 7f137dc80640 10 monclient: get_auth_request con 0x7f135c06e0f0 auth_method 0 2024-03-25T18:13:31.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.477+0000 7f136e7fc640 10 client.4441.objecter ms_handle_connect 0x7f135c06e0f0 2024-03-25T18:13:31.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.481+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f1368053510 con 0x7f13780b4d60 2024-03-25T18:13:31.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.481+0000 7f136e7fc640 10 client.4441.objecter ms_dispatch 0x7f1378000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:31.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.481+0000 7f136e7fc640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:31.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.481+0000 7f136e7fc640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:31.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.481+0000 7f136e7fc640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:13:31.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.482+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f136804f020 con 0x7f13780b4d60 2024-03-25T18:13:31.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.482+0000 7f136e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.482+0000 7f136e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:31.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.717+0000 7f137fee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:31.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.717+0000 7f137fee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:31.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.717+0000 7f137fee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:13:31.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.717+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1378141ee0 con 0x7f13780b4d60 2024-03-25T18:13:31.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.760+0000 7f136e7fc640 1 -- 172.21.15.46:0/901079929 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f1368050f50 con 0x7f13780b4d60 2024-03-25T18:13:31.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.760+0000 7f136e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:31.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.760+0000 7f136e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:31.762 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:31.762 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 10 librados: watch_flush enter 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 10 librados: watch_flush exit 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 >> v1:172.21.15.164:6800/1612084612 conn(0x7f135c06e0f0 legacy=0x7f135c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 10 monclient: shutdown 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 >> v1:172.21.15.164:6789/0 conn(0x7f13780b4d60 legacy=0x7f1378132e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.762+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 shutdown_connections 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.763+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 >> 172.21.15.46:0/901079929 conn(0x7f13780617a0 msgr2=0x7f13780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.763+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 shutdown_connections 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.763+0000 7f137fee4640 1 -- 172.21.15.46:0/901079929 wait complete. 2024-03-25T18:13:31.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:31.763+0000 7f137fee4640 1 librados: shutdown 2024-03-25T18:13:31.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:38.583 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:38.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.698+0000 7f2fa3ee4640 10 monclient: get_monmap_and_config 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.698+0000 7f2fa3ee4640 10 monclient: build_initial_monmap 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.698+0000 7f2fa3ee4640 10 monclient: monmap: 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:38.699783+0000 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:38.699783+0000 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:38.700 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:38.701 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:38.701 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.698+0000 7f2fa3ee4640 1 Processor -- start 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 1 -- start start 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient: init 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c142520 con 0x7f2f9c1430c0 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0b28a0 con 0x7f2f9c13f710 2024-03-25T18:13:38.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0b2a30 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:38.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.699+0000 7f2fa3ee4640 10 monclient(hunting): authenticate will time out at 7527.962435s 2024-03-25T18:13:38.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.712+0000 7f2fa147f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2f9c13f710 0x7f2f9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53306/0 (socket says 172.21.15.46:53306) 2024-03-25T18:13:38.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.712+0000 7f2fa147f640 1 -- 172.21.15.46:0/4092207404 learned_addr learned my addr 172.21.15.46:0/4092207404 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.716+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524745131 0 0) 0x7f2f9c0b28a0 con 0x7f2f9c13f710 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.716+0000 7f2fa0c7e640 10 monclient(hunting): _negotiate 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.716+0000 7f2fa0c7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.716+0000 7f2fa0c7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.717+0000 7f2fa0c7e640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:13:38.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.717+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f88003520 con 0x7f2f9c13f710 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3375913221 0 0) 0x7f2f88003520 con 0x7f2f9c13f710 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 >> v1:172.21.15.46:6790/0 conn(0x7f2f9c0a98f0 legacy=0x7f2f9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 >> v1:172.21.15.46:6789/0 conn(0x7f2f9c1430c0 legacy=0x7f2f9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: _finish_hunting 0 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 monclient: found mon.noname-c 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0b44c0 con 0x7f2f9c13f710 2024-03-25T18:13:38.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: _finish_auth 0 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: _check_auth_tickets 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f8c001690 con 0x7f2f9c13f710 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa0c7e640 10 monclient: dump: 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:38.737 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:38.738 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa3ee4640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:13:38.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.734+0000 7f2fa3ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:38.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f8c0017f0 con 0x7f2f9c13f710 2024-03-25T18:13:38.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:38.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 1 -- 172.21.15.46:0/4092207404 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f8c001c00 con 0x7f2f9c13f710 2024-03-25T18:13:38.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa3ee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa3ee4640 4 set_mon_vals no callback set 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.749+0000 7f2fa0c7e640 10 monclient: dump: 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:38.752 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 10 monclient: shutdown 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4092207404 >> v1:172.21.15.164:6789/0 conn(0x7f2f9c13f710 legacy=0x7f2f9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4092207404 shutdown_connections 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4092207404 >> 172.21.15.46:0/4092207404 conn(0x7f2f9c0b7ea0 msgr2=0x7f2f9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:38.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.750+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4092207404 shutdown_connections 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.751+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4092207404 wait complete. 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.751+0000 7f2fa3ee4640 10 monclient: build_initial_monmap 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.751+0000 7f2fa3ee4640 10 monclient: monmap: 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 librados: starting msgr at 2024-03-25T18:13:38.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 librados: starting objecter 2024-03-25T18:13:38.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 Processor -- start 2024-03-25T18:13:38.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 -- start start 2024-03-25T18:13:38.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 librados: setting wanted keys 2024-03-25T18:13:38.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 1 librados: calling monclient init 2024-03-25T18:13:38.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.752+0000 7f2fa3ee4640 10 monclient: init 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c13be70 con 0x7f2f9c1430c0 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c131660 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c1317f0 con 0x7f2f9c13f710 2024-03-25T18:13:38.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:38.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.753+0000 7f2fa3ee4640 10 monclient(hunting): authenticate will time out at 7528.015975s 2024-03-25T18:13:38.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.766+0000 7f2fa147f640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f2f9c13f710 0x7f2f9c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:46240/0 (socket says 172.21.15.46:46240) 2024-03-25T18:13:38.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.766+0000 7f2fa147f640 1 -- 172.21.15.46:0/4125437722 learned_addr learned my addr 172.21.15.46:0/4125437722 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:38.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.767+0000 7f2f927fc640 10 client.?.objecter ms_handle_connect 0x7f2f9c13f710 2024-03-25T18:13:38.769 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.767+0000 7f2f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:38.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.774+0000 7f2f927fc640 10 client.?.objecter ms_handle_connect 0x7f2f9c0a98f0 2024-03-25T18:13:38.775 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.774+0000 7f2f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:38.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.775+0000 7f2f927fc640 10 client.?.objecter ms_handle_connect 0x7f2f9c1430c0 2024-03-25T18:13:38.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.775+0000 7f2f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126612542 0 0) 0x7f2f9c1317f0 con 0x7f2f9c13f710 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:13:38.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.779+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80004eb0 con 0x7f2f9c13f710 2024-03-25T18:13:38.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.872+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813925969 0 0) 0x7f2f80004eb0 con 0x7f2f9c13f710 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f9c1317f0 con 0x7f2f9c13f710 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f8c001c00 con 0x7f2f9c13f710 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 10 monclient(hunting): dump: 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.875 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:38.876 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:38.876 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:38.876 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.873+0000 7f2f927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419744305 0 0) 0x7f2f9c131660 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:13:38.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.880+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80004eb0 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314821168 0 0) 0x7f2f9c13be70 con 0x7f2f9c1430c0 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:13:38.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.886+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c131660 con 0x7f2f9c1430c0 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414887450 0 0) 0x7f2f80004eb0 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f9c13be70 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f98001830 con 0x7f2f9c0a98f0 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.895+0000 7f2f927fc640 10 monclient(hunting): dump: 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.897 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:38.898 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3599983120 0 0) 0x7f2f9c1317f0 con 0x7f2f9c13f710 2024-03-25T18:13:38.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 >> v1:172.21.15.164:6789/0 conn(0x7f2f9c0a98f0 legacy=0x7f2f9c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 >> v1:172.21.15.46:6789/0 conn(0x7f2f9c1430c0 legacy=0x7f2f9c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 1 monclient: found mon.c 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c131980 con 0x7f2f9c13f710 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2f927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 10 monclient: _renew_subs 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c135c60 con 0x7f2f9c13f710 2024-03-25T18:13:38.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:38.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 10 monclient: _renew_subs 2024-03-25T18:13:38.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:38.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c0b7770 con 0x7f2f9c13f710 2024-03-25T18:13:38.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.902+0000 7f2fa3ee4640 1 librados: init done 2024-03-25T18:13:38.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.903+0000 7f2fa3ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.903+0000 7f2fa3ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.903+0000 7f2fa3ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:38.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.903+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c1371d0 con 0x7f2f9c13f710 2024-03-25T18:13:38.919 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f8c004a90 con 0x7f2f9c13f710 2024-03-25T18:13:38.919 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:38.919 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2f8c004e60 con 0x7f2f9c13f710 2024-03-25T18:13:38.919 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f927fc640 10 monclient: dump: 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:38.920 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:38.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:38.918+0000 7f2f93fff640 4 set_mon_vals no callback set 2024-03-25T18:13:39.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.253+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f2f8c004fc0 con 0x7f2f9c13f710 2024-03-25T18:13:39.255 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.254+0000 7f2fa1c80640 10 monclient: get_auth_request con 0x7f2f8006e0b0 auth_method 0 2024-03-25T18:13:39.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.254+0000 7f2f927fc640 10 client.4454.objecter ms_handle_connect 0x7f2f8006e0b0 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f2f8c054970 con 0x7f2f9c13f710 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 10 client.4454.objecter ms_dispatch 0x7f2f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f8c050480 con 0x7f2f9c13f710 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.260+0000 7f2f927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:39.402 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:15.319 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.413 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:15.331 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.455 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:15.372 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.475 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:15.346 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.476 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:15.393 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.479 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:15.351 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.483 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:15.401 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.486+0000 7f2fa3ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.486+0000 7f2fa3ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.486+0000 7f2fa3ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:39.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.486+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f9c074cc0 con 0x7f2f9c13f710 2024-03-25T18:13:39.506 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:15.378 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.520 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:15.438 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:39.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.548+0000 7f2f927fc640 1 -- 172.21.15.46:0/4125437722 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f2f8c0523b0 con 0x7f2f9c13f710 2024-03-25T18:13:39.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.548+0000 7f2f927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.548+0000 7f2f927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:39.550 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:39.550 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:39.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 10 librados: watch_flush enter 2024-03-25T18:13:39.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 10 librados: watch_flush exit 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 10 client.4454.objecter successfully canceled tick 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 >> v1:172.21.15.164:6800/1612084612 conn(0x7f2f8006e0b0 legacy=0x7f2f80070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 10 monclient: shutdown 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 >> v1:172.21.15.46:6790/0 conn(0x7f2f9c13f710 legacy=0x7f2f9c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 shutdown_connections 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.550+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 >> 172.21.15.46:0/4125437722 conn(0x7f2f9c0b7ea0 msgr2=0x7f2f9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.551+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 shutdown_connections 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.551+0000 7f2fa3ee4640 1 -- 172.21.15.46:0/4125437722 wait complete. 2024-03-25T18:13:39.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:39.551+0000 7f2fa3ee4640 1 librados: shutdown 2024-03-25T18:13:39.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:46.368 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:46.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.483+0000 7f797d9f6640 10 monclient: get_monmap_and_config 2024-03-25T18:13:46.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.483+0000 7f797d9f6640 10 monclient: build_initial_monmap 2024-03-25T18:13:46.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.483+0000 7f797d9f6640 10 monclient: monmap: 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:46.485306+0000 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:46.485306+0000 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:46.486 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.484+0000 7f797d9f6640 1 Processor -- start 2024-03-25T18:13:46.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.484+0000 7f797d9f6640 1 -- start start 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.484+0000 7f797d9f6640 10 monclient: init 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f79780a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f797813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f79781430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7978142520 con 0x7f79781430c0 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780b28a0 con 0x7f79780a98f0 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780b2a30 con 0x7f797813f710 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:46.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.485+0000 7f797d9f6640 10 monclient(hunting): authenticate will time out at 7535.747971s 2024-03-25T18:13:46.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.500+0000 7f79777fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f79781430c0 0x7f797814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60734/0 (socket says 172.21.15.46:60734) 2024-03-25T18:13:46.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.500+0000 7f79777fe640 1 -- 172.21.15.46:0/2504100192 learned_addr learned my addr 172.21.15.46:0/2504100192 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454827077 0 0) 0x7f79780b28a0 con 0x7f79780a98f0 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:13:46.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.517+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7960003520 con 0x7f79780a98f0 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342419697 0 0) 0x7f7978142520 con 0x7f79781430c0 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:13:46.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.519+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79780b28a0 con 0x7f79781430c0 2024-03-25T18:13:46.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221622436 0 0) 0x7f79780b28a0 con 0x7f79781430c0 2024-03-25T18:13:46.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 >> v1:172.21.15.46:6790/0 conn(0x7f797813f710 legacy=0x7f79781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 >> v1:172.21.15.164:6789/0 conn(0x7f79780a98f0 legacy=0x7f79780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 1 monclient: found mon.noname-a 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780b44c0 con 0x7f79781430c0 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f7975ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f797d9f6640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:13:46.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.529+0000 7f797d9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.531+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f79700019b0 con 0x7f79781430c0 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.531+0000 7f7975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.531+0000 7f7975ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.531+0000 7f7975ffb640 10 monclient: dump: 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.533 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:46.534 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:46.534 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:46.534 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:46.534 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.531+0000 7f797d9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:46.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7970001c40 con 0x7f79781430c0 2024-03-25T18:13:46.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:46.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 1 -- 172.21.15.46:0/2504100192 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7970001f50 con 0x7f79781430c0 2024-03-25T18:13:46.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:46.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 4 set_mon_vals no callback set 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f7975ffb640 10 monclient: dump: 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:46.541 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 10 monclient: shutdown 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 1 -- 172.21.15.46:0/2504100192 >> v1:172.21.15.46:6789/0 conn(0x7f79781430c0 legacy=0x7f797814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 1 -- 172.21.15.46:0/2504100192 shutdown_connections 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 1 -- 172.21.15.46:0/2504100192 >> 172.21.15.46:0/2504100192 conn(0x7f79780b7ea0 msgr2=0x7f79780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.539+0000 7f797d9f6640 1 -- 172.21.15.46:0/2504100192 shutdown_connections 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.540+0000 7f797d9f6640 1 -- 172.21.15.46:0/2504100192 wait complete. 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.540+0000 7f797d9f6640 10 monclient: build_initial_monmap 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.540+0000 7f797d9f6640 10 monclient: monmap: 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:46.543 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.540+0000 7f797d9f6640 1 librados: starting msgr at 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.540+0000 7f797d9f6640 1 librados: starting objecter 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 1 Processor -- start 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 1 -- start start 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 1 librados: setting wanted keys 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 1 librados: calling monclient init 2024-03-25T18:13:46.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient: init 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f79780a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f797813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f79781430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:46.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.541+0000 7f797d9f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780abb00 con 0x7f797813f710 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.542+0000 7f797d9f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79781d76a0 con 0x7f79780a98f0 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.542+0000 7f797d9f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79781d82d0 con 0x7f79781430c0 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.542+0000 7f797d9f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.542+0000 7f797d9f6640 10 monclient(hunting): authenticate will time out at 7535.804576s 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.545+0000 7f7976ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f79780a98f0 0x7f79781379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:54876/0 (socket says 172.21.15.46:54876) 2024-03-25T18:13:46.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.545+0000 7f7976ffd640 1 -- 172.21.15.46:0/4023512315 learned_addr learned my addr 172.21.15.46:0/4023512315 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:46.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.546+0000 7f795ffff640 10 client.?.objecter ms_handle_connect 0x7f79780a98f0 2024-03-25T18:13:46.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.546+0000 7f795ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.548+0000 7f795ffff640 10 client.?.objecter ms_handle_connect 0x7f79781430c0 2024-03-25T18:13:46.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.548+0000 7f795ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763919871 0 0) 0x7f79781d82d0 con 0x7f79781430c0 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:13:46.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.552+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964004eb0 con 0x7f79781430c0 2024-03-25T18:13:46.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.554+0000 7f795ffff640 10 client.?.objecter ms_handle_connect 0x7f797813f710 2024-03-25T18:13:46.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.554+0000 7f795ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801606272 0 0) 0x7f79781d76a0 con 0x7f79780a98f0 2024-03-25T18:13:46.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:13:46.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.558+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79781d82d0 con 0x7f79780a98f0 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2967585632 0 0) 0x7f7964004eb0 con 0x7f79781430c0 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79781d76a0 con 0x7f79781430c0 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7970001f50 con 0x7f79781430c0 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 10 monclient(hunting): dump: 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.565 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:46.566 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:46.566 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:46.566 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:46.566 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.566 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.563+0000 7f795ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:46.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684288542 0 0) 0x7f79780abb00 con 0x7f797813f710 2024-03-25T18:13:46.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:13:46.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.574+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964004eb0 con 0x7f797813f710 2024-03-25T18:13:46.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4163096232 0 0) 0x7f79781d76a0 con 0x7f79781430c0 2024-03-25T18:13:46.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 >> v1:172.21.15.164:6789/0 conn(0x7f79780a98f0 legacy=0x7f79781379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 >> v1:172.21.15.46:6789/0 conn(0x7f797813f710 legacy=0x7f79781d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 1 monclient: found mon.c 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79781d8460 con 0x7f79781430c0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f795ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 10 monclient: _renew_subs 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978134e50 con 0x7f79781430c0 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:46.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 10 monclient: _renew_subs 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781351a0 con 0x7f79781430c0 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.577+0000 7f797d9f6640 1 librados: init done 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.578+0000 7f797d9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.578+0000 7f797d9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.578+0000 7f797d9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:46.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.578+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79781d7500 con 0x7f79781430c0 2024-03-25T18:13:46.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7970004e10 con 0x7f79781430c0 2024-03-25T18:13:46.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:46.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7970004f80 con 0x7f79781430c0 2024-03-25T18:13:46.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f795ffff640 10 monclient: dump: 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:46.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.897+0000 7f79757fa640 4 set_mon_vals no callback set 2024-03-25T18:13:46.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.898+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f797001af50 con 0x7f79781430c0 2024-03-25T18:13:46.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.899+0000 7f7976ffd640 10 monclient: get_auth_request con 0x7f796406de00 auth_method 0 2024-03-25T18:13:46.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.899+0000 7f795ffff640 10 client.4460.objecter ms_handle_connect 0x7f796406de00 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f79700549b0 con 0x7f79781430c0 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 10 client.4460.objecter ms_dispatch 0x7f7978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79700504c0 con 0x7f79781430c0 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:46.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:46.904+0000 7f795ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:47.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.146+0000 7f797d9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.146+0000 7f797d9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.146+0000 7f797d9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:47.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.146+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79780627b0 con 0x7f79781430c0 2024-03-25T18:13:47.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.174+0000 7f795ffff640 1 -- 172.21.15.46:0/4023512315 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f79700523f0 con 0x7f79781430c0 2024-03-25T18:13:47.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.174+0000 7f795ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.174+0000 7f795ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:47.176 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:47.176 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 10 librados: watch_flush enter 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 10 librados: watch_flush exit 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 10 client.4460.objecter successfully canceled tick 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 >> v1:172.21.15.164:6800/1612084612 conn(0x7f796406de00 legacy=0x7f79640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 10 monclient: shutdown 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 >> v1:172.21.15.46:6790/0 conn(0x7f79781430c0 legacy=0x7f7978133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 shutdown_connections 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 >> 172.21.15.46:0/4023512315 conn(0x7f79780b7ea0 msgr2=0x7f7978074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 shutdown_connections 2024-03-25T18:13:47.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.176+0000 7f797d9f6640 1 -- 172.21.15.46:0/4023512315 wait complete. 2024-03-25T18:13:47.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:47.177+0000 7f797d9f6640 1 librados: shutdown 2024-03-25T18:13:47.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:53.997 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.112+0000 7f904a074640 10 monclient: get_monmap_and_config 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.112+0000 7f904a074640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.112+0000 7f904a074640 10 monclient: monmap: 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:13:54.113365+0000 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:13:54.113365+0000 2024-03-25T18:13:54.113 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.114 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.114 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:54.114 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:13:54.114 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:13:54.114 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.113+0000 7f904a074640 1 Processor -- start 2024-03-25T18:13:54.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.113+0000 7f904a074640 1 -- start start 2024-03-25T18:13:54.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.113+0000 7f904a074640 10 monclient: init 2024-03-25T18:13:54.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient(hunting): picked mon.noname-b con 0x7f904406c380 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient(hunting): picked mon.noname-c con 0x7f90440b12d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient(hunting): picked mon.noname-a con 0x7f90440aedf0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440b16f0 con 0x7f90440aedf0 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440adfe0 con 0x7f90440b12d0 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440add00 con 0x7f904406c380 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.114+0000 7f904a074640 10 monclient(hunting): authenticate will time out at 7543.376003s 2024-03-25T18:13:54.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.125+0000 7f9042ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f90440b12d0 0x7f90440ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45718/0 (socket says 172.21.15.46:45718) 2024-03-25T18:13:54.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.125+0000 7f9042ffd640 1 -- 172.21.15.46:0/3844456133 learned_addr learned my addr 172.21.15.46:0/3844456133 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085012538 0 0) 0x7f90440adfe0 con 0x7f90440b12d0 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:13:54.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.138+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c003520 con 0x7f90440b12d0 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272716584 0 0) 0x7f90440add00 con 0x7f904406c380 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:13:54.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.143+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90440adfe0 con 0x7f904406c380 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203491206 0 0) 0x7f90440b16f0 con 0x7f90440aedf0 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:13:54.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.146+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90440add00 con 0x7f90440aedf0 2024-03-25T18:13:54.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3071108918 0 0) 0x7f90440adfe0 con 0x7f904406c380 2024-03-25T18:13:54.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 >> v1:172.21.15.164:6789/0 conn(0x7f90440b12d0 legacy=0x7f90440ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 >> v1:172.21.15.46:6789/0 conn(0x7f90440aedf0 legacy=0x7f904414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 monclient: found mon.noname-b 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440ae2c0 con 0x7f904406c380 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9034001670 con 0x7f904406c380 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f904a074640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f904a074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f90427fc640 10 monclient: dump: 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.155 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.153+0000 7f904a074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90340017d0 con 0x7f904406c380 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 1 -- 172.21.15.46:0/3844456133 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9034001b60 con 0x7f904406c380 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f904a074640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f90427fc640 10 monclient: dump: 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.172 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.171+0000 7f904a074640 4 set_mon_vals no callback set 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 10 monclient: shutdown 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 1 -- 172.21.15.46:0/3844456133 >> v1:172.21.15.46:6790/0 conn(0x7f904406c380 legacy=0x7f90440b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 1 -- 172.21.15.46:0/3844456133 shutdown_connections 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 1 -- 172.21.15.46:0/3844456133 >> 172.21.15.46:0/3844456133 conn(0x7f90440b5d10 msgr2=0x7f90440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 1 -- 172.21.15.46:0/3844456133 shutdown_connections 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.172+0000 7f904a074640 1 -- 172.21.15.46:0/3844456133 wait complete. 2024-03-25T18:13:54.175 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 10 monclient: monmap: 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 librados: starting msgr at 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 librados: starting objecter 2024-03-25T18:13:54.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 Processor -- start 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 -- start start 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 librados: setting wanted keys 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 1 librados: calling monclient init 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.173+0000 7f904a074640 10 monclient: init 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient(hunting): picked mon.noname-b con 0x7f904406c380 addr v1:172.21.15.164:6789/0 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient(hunting): picked mon.noname-a con 0x7f90440aedf0 addr v1:172.21.15.46:6789/0 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient(hunting): picked mon.noname-c con 0x7f90440b12d0 addr v1:172.21.15.46:6790/0 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904406c8e0 con 0x7f90440aedf0 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90441c34c0 con 0x7f904406c380 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90441c30e0 con 0x7f90440b12d0 2024-03-25T18:13:54.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.174+0000 7f904a074640 10 monclient(hunting): authenticate will time out at 7543.436040s 2024-03-25T18:13:54.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.179+0000 7f90437fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f904406c380 0x7f90441db1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:45778/0 (socket says 172.21.15.46:45778) 2024-03-25T18:13:54.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.179+0000 7f90437fe640 1 -- 172.21.15.46:0/2041078754 learned_addr learned my addr 172.21.15.46:0/2041078754 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:13:54.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.186+0000 7f902bfff640 10 client.?.objecter ms_handle_connect 0x7f90440b12d0 2024-03-25T18:13:54.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.186+0000 7f902bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.187+0000 7f902bfff640 10 client.?.objecter ms_handle_connect 0x7f904406c380 2024-03-25T18:13:54.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.187+0000 7f902bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.190+0000 7f902bfff640 10 client.?.objecter ms_handle_connect 0x7f90440aedf0 2024-03-25T18:13:54.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.190+0000 7f902bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013965332 0 0) 0x7f904406c8e0 con 0x7f90440aedf0 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:13:54.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.199+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9030004eb0 con 0x7f90440aedf0 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800622924 0 0) 0x7f90441c34c0 con 0x7f904406c380 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:13:54.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.200+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904406c8e0 con 0x7f904406c380 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062502185 0 0) 0x7f90441c30e0 con 0x7f90440b12d0 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:13:54.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.206+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90441c34c0 con 0x7f90440b12d0 2024-03-25T18:13:54.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151141414 0 0) 0x7f90441c34c0 con 0x7f90440b12d0 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90441c30e0 con 0x7f90440b12d0 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f903c001c70 con 0x7f90440b12d0 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 10 monclient(hunting): dump: 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.220 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.221 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.221 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.219+0000 7f902bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120000990 0 0) 0x7f9030004eb0 con 0x7f90440aedf0 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90441c34c0 con 0x7f90440aedf0 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9038001770 con 0x7f90440aedf0 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.226+0000 7f902bfff640 10 monclient(hunting): dump: 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.227 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.228 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.228 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.228 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.228 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.228 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390995956 0 0) 0x7f904406c8e0 con 0x7f904406c380 2024-03-25T18:13:54.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9030004eb0 con 0x7f904406c380 2024-03-25T18:13:54.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9034001ae0 con 0x7f904406c380 2024-03-25T18:13:54.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.231+0000 7f902bfff640 10 monclient(hunting): dump: 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.233 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 10139718 0 0) 0x7f90441c30e0 con 0x7f90440b12d0 2024-03-25T18:13:54.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 >> v1:172.21.15.164:6789/0 conn(0x7f904406c380 legacy=0x7f90441db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 >> v1:172.21.15.46:6789/0 conn(0x7f90440aedf0 legacy=0x7f90441c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 1 monclient: found mon.c 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90441c3270 con 0x7f90440b12d0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f902bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 10 monclient: _renew_subs 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90441c18e0 con 0x7f90440b12d0 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:54.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 10 monclient: _renew_subs 2024-03-25T18:13:54.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90441dad40 con 0x7f90440b12d0 2024-03-25T18:13:54.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.235+0000 7f904a074640 1 librados: init done 2024-03-25T18:13:54.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.236+0000 7f904a074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.238 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.236+0000 7f904a074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.236+0000 7f904a074640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.236+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90441c3650 con 0x7f90440b12d0 2024-03-25T18:13:54.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.448+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f903c002080 con 0x7f90440b12d0 2024-03-25T18:13:54.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.448+0000 7f902bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.448+0000 7f9041ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.554+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f903c003b30 con 0x7f90440b12d0 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.554+0000 7f902bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.554+0000 7f902bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.554+0000 7f902bfff640 10 monclient: dump: 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:13:54.555 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:13:54.556 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:13:54.556 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:13:54.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.555+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f903c019b00 con 0x7f90440b12d0 2024-03-25T18:13:54.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.555+0000 7f90437fe640 10 monclient: get_auth_request con 0x7f903006e0f0 auth_method 0 2024-03-25T18:13:54.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.556+0000 7f902bfff640 10 client.4466.objecter ms_handle_connect 0x7f903006e0f0 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f903c053890 con 0x7f90440b12d0 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 10 client.4466.objecter ms_dispatch 0x7f9044000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f903c04f1d0 con 0x7f90440b12d0 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.564+0000 7f902bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:54.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.779+0000 7f904a074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.779+0000 7f904a074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.779+0000 7f904a074640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:13:54.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.779+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90440627b0 con 0x7f90440b12d0 2024-03-25T18:13:54.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.815+0000 7f902bfff640 1 -- 172.21.15.46:0/2041078754 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f903c0512d0 con 0x7f90440b12d0 2024-03-25T18:13:54.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.815+0000 7f902bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.815+0000 7f902bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:54.816 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:13:54.816 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 10 librados: watch_flush enter 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 10 librados: watch_flush exit 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 >> v1:172.21.15.164:6800/1612084612 conn(0x7f903006e0f0 legacy=0x7f90300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 10 monclient: shutdown 2024-03-25T18:13:54.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 >> v1:172.21.15.46:6790/0 conn(0x7f90440b12d0 legacy=0x7f90441e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 shutdown_connections 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 >> 172.21.15.46:0/2041078754 conn(0x7f90440b5d10 msgr2=0x7f90440af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 shutdown_connections 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.818+0000 7f904a074640 1 -- 172.21.15.46:0/2041078754 wait complete. 2024-03-25T18:13:54.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:13:54.819+0000 7f904a074640 1 librados: shutdown 2024-03-25T18:13:54.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:59.694 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:35.611 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.700 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:35.618 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.748 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:35.666 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.772 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:35.690 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.776 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:35.694 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.777 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:35.648 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.783 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:35.654 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.799 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:35.716 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:59.805 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:35.676 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:01.638 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.754+0000 7f70c1820640 10 monclient: get_monmap_and_config 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.754+0000 7f70c1820640 10 monclient: build_initial_monmap 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.754+0000 7f70c1820640 10 monclient: monmap: 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:01.754882+0000 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:01.754882+0000 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:01.755 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:01.756 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:01.756 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:01.756 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:01.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.754+0000 7f70c1820640 1 Processor -- start 2024-03-25T18:14:01.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 1 -- start start 2024-03-25T18:14:01.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient: init 2024-03-25T18:14:01.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:01.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-b con 0x7f70bc1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc142520 con 0x7f70bc13f710 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0b28a0 con 0x7f70bc0a98f0 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0b2a30 con 0x7f70bc1430c0 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.755+0000 7f70c1820640 10 monclient(hunting): authenticate will time out at 7551.017193s 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.756+0000 7f70baffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f70bc0a98f0 0x7f70bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50172/0 (socket says 172.21.15.46:50172) 2024-03-25T18:14:01.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.756+0000 7f70baffd640 1 -- 172.21.15.46:0/2556928382 learned_addr learned my addr 172.21.15.46:0/2556928382 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:01.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.778+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141899084 0 0) 0x7f70bc0b2a30 con 0x7f70bc1430c0 2024-03-25T18:14:01.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.779+0000 7f70ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.779+0000 7f70ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.779+0000 7f70ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.779+0000 7f70ba7fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:14:01.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.779+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a8003520 con 0x7f70bc1430c0 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879647000 0 0) 0x7f70a8003520 con 0x7f70bc1430c0 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 >> v1:172.21.15.164:6789/0 conn(0x7f70bc0a98f0 legacy=0x7f70bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 >> v1:172.21.15.46:6789/0 conn(0x7f70bc13f710 legacy=0x7f70bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 monclient: found mon.noname-b 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:14:01.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc0b44c0 con 0x7f70bc1430c0 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f70b4001a10 con 0x7f70bc1430c0 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70ba7fc640 10 monclient: dump: 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:01.791 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:01.792 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:01.792 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:01.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70c1820640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:14:01.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.789+0000 7f70c1820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b4001b70 con 0x7f70bc1430c0 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 1 -- 172.21.15.46:0/2556928382 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f70b4001f80 con 0x7f70bc1430c0 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:01.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70ba7fc640 10 monclient: dump: 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:01.796 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:01.797 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:01.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70c1820640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:01.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.794+0000 7f70c1820640 4 set_mon_vals no callback set 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 10 monclient: shutdown 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 1 -- 172.21.15.46:0/2556928382 >> v1:172.21.15.46:6790/0 conn(0x7f70bc1430c0 legacy=0x7f70bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 1 -- 172.21.15.46:0/2556928382 shutdown_connections 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 1 -- 172.21.15.46:0/2556928382 >> 172.21.15.46:0/2556928382 conn(0x7f70bc0b7ea0 msgr2=0x7f70bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 1 -- 172.21.15.46:0/2556928382 shutdown_connections 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.795+0000 7f70c1820640 1 -- 172.21.15.46:0/2556928382 wait complete. 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 10 monclient: build_initial_monmap 2024-03-25T18:14:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 10 monclient: monmap: 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 librados: starting msgr at 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 librados: starting objecter 2024-03-25T18:14:01.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 Processor -- start 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 -- start start 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 librados: setting wanted keys 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 1 librados: calling monclient init 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.796+0000 7f70c1820640 10 monclient: init 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-b con 0x7f70bc0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc136ec0 con 0x7f70bc13f710 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc136c20 con 0x7f70bc0a98f0 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc137db0 con 0x7f70bc1430c0 2024-03-25T18:14:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.797+0000 7f70c1820640 10 monclient(hunting): authenticate will time out at 7551.059121s 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70baffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f70bc0a98f0 0x7f70bc1376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50202/0 (socket says 172.21.15.46:50202) 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70baffd640 1 -- 172.21.15.46:0/1286506898 learned_addr learned my addr 172.21.15.46:0/1286506898 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70b8ff9640 10 client.?.objecter ms_handle_connect 0x7f70bc0a98f0 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70b8ff9640 10 client.?.objecter ms_handle_connect 0x7f70bc13f710 2024-03-25T18:14:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.798+0000 7f70b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.876+0000 7f70b8ff9640 10 client.?.objecter ms_handle_connect 0x7f70bc1430c0 2024-03-25T18:14:01.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.876+0000 7f70b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207203903 0 0) 0x7f70bc137db0 con 0x7f70bc1430c0 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:14:01.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.884+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c004eb0 con 0x7f70bc1430c0 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2739938686 0 0) 0x7f709c004eb0 con 0x7f70bc1430c0 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70bc137db0 con 0x7f70bc1430c0 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f70b4001e80 con 0x7f70bc1430c0 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:01.903 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.902+0000 7f70b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305063494 0 0) 0x7f70bc136c20 con 0x7f70bc0a98f0 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:14:01.918 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.917+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c004eb0 con 0x7f70bc0a98f0 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 28586393 0 0) 0x7f70bc137db0 con 0x7f70bc1430c0 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 >> v1:172.21.15.164:6789/0 conn(0x7f70bc0a98f0 legacy=0x7f70bc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 >> v1:172.21.15.46:6789/0 conn(0x7f70bc13f710 legacy=0x7f70bc0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 1 monclient: found mon.c 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc138b50 con 0x7f70bc1430c0 2024-03-25T18:14:01.920 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 5 monclient: authenticate success, global_id 4472 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 10 monclient: _renew_subs 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc139830 con 0x7f70bc1430c0 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 10 monclient: _renew_subs 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc1369b0 con 0x7f70bc1430c0 2024-03-25T18:14:01.921 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.919+0000 7f70c1820640 1 librados: init done 2024-03-25T18:14:01.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.920+0000 7f70c1820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:01.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.920+0000 7f70c1820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:01.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.920+0000 7f70c1820640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:01.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:01.920+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc139a20 con 0x7f70bc1430c0 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b4003c20 con 0x7f70bc1430c0 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f70b4003d90 con 0x7f70bc1430c0 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b8ff9640 10 monclient: dump: 2024-03-25T18:14:02.251 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:02.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.250+0000 7f70b9ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:02.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.251+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f70b4019d60 con 0x7f70bc1430c0 2024-03-25T18:14:02.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.252+0000 7f70baffd640 10 monclient: get_auth_request con 0x7f709c06dce0 auth_method 0 2024-03-25T18:14:02.253 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.252+0000 7f70b8ff9640 10 client.4472.objecter ms_handle_connect 0x7f709c06dce0 2024-03-25T18:14:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f70b40537f0 con 0x7f70bc1430c0 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 10 client.4472.objecter ms_dispatch 0x7f70bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70b404f300 con 0x7f70bc1430c0 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.258+0000 7f70b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:02.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.517+0000 7f70c1820640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.517+0000 7f70c1820640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.517+0000 7f70c1820640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:02.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.517+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70bc0627b0 con 0x7f70bc1430c0 2024-03-25T18:14:02.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.546+0000 7f70b8ff9640 1 -- 172.21.15.46:0/1286506898 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f70b4051230 con 0x7f70bc1430c0 2024-03-25T18:14:02.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.546+0000 7f70b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.546+0000 7f70b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:02.547 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:02.547 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:02.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 10 librados: watch_flush enter 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 10 librados: watch_flush exit 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 10 client.4472.objecter successfully canceled tick 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 >> v1:172.21.15.164:6800/1612084612 conn(0x7f709c06dce0 legacy=0x7f709c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 10 monclient: shutdown 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 >> v1:172.21.15.46:6790/0 conn(0x7f70bc1430c0 legacy=0x7f70bc1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 shutdown_connections 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 >> 172.21.15.46:0/1286506898 conn(0x7f70bc0b7ea0 msgr2=0x7f70bc13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 shutdown_connections 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 -- 172.21.15.46:0/1286506898 wait complete. 2024-03-25T18:14:02.550 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:02.548+0000 7f70c1820640 1 librados: shutdown 2024-03-25T18:14:02.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:09.368 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:09.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.471+0000 7f6a13715640 10 monclient: get_monmap_and_config 2024-03-25T18:14:09.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.471+0000 7f6a13715640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.472+0000 7f6a13715640 10 monclient: monmap: 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:09.472657+0000 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:09.472657+0000 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:09.473 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.472+0000 7f6a13715640 1 Processor -- start 2024-03-25T18:14:09.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.472+0000 7f6a13715640 1 -- start start 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.472+0000 7f6a13715640 10 monclient: init 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c1430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c142520 con 0x7f6a0c0a98f0 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0b28a0 con 0x7f6a0c1430c0 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0b2a30 con 0x7f6a0c13f710 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.473+0000 7f6a13715640 10 monclient(hunting): authenticate will time out at 7558.735154s 2024-03-25T18:14:09.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.484+0000 7f6a11cb2640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6a0c1430c0 0x7f6a0c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36762/0 (socket says 172.21.15.46:36762) 2024-03-25T18:14:09.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.484+0000 7f6a11cb2640 1 -- 172.21.15.46:0/635196713 learned_addr learned my addr 172.21.15.46:0/635196713 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566816717 0 0) 0x7f6a0c0b28a0 con 0x7f6a0c1430c0 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:14:09.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.491+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8003520 con 0x7f6a0c1430c0 2024-03-25T18:14:09.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487005381 0 0) 0x7f69f8003520 con 0x7f6a0c1430c0 2024-03-25T18:14:09.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 >> v1:172.21.15.46:6790/0 conn(0x7f6a0c13f710 legacy=0x7f6a0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 >> v1:172.21.15.46:6789/0 conn(0x7f6a0c0a98f0 legacy=0x7f6a0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 monclient: found mon.noname-c 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c0b44c0 con 0x7f6a0c1430c0 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6a04001a10 con 0x7f6a0c1430c0 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a03fff640 10 monclient: dump: 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.503 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a13715640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:14:09.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.501+0000 7f6a13715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a04001b70 con 0x7f6a0c1430c0 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 1 -- 172.21.15.46:0/635196713 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6a04001f80 con 0x7f6a0c1430c0 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a03fff640 10 monclient: dump: 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a13715640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:09.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.515+0000 7f6a13715640 4 set_mon_vals no callback set 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.516+0000 7f6a13715640 10 monclient: shutdown 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.516+0000 7f6a13715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.516+0000 7f6a13715640 1 -- 172.21.15.46:0/635196713 >> v1:172.21.15.164:6789/0 conn(0x7f6a0c1430c0 legacy=0x7f6a0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.516+0000 7f6a13715640 1 -- 172.21.15.46:0/635196713 shutdown_connections 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.516+0000 7f6a13715640 1 -- 172.21.15.46:0/635196713 >> 172.21.15.46:0/635196713 conn(0x7f6a0c0b7ea0 msgr2=0x7f6a0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.517+0000 7f6a13715640 1 -- 172.21.15.46:0/635196713 shutdown_connections 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.517+0000 7f6a13715640 1 -- 172.21.15.46:0/635196713 wait complete. 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.517+0000 7f6a13715640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.517+0000 7f6a13715640 10 monclient: monmap: 2024-03-25T18:14:09.519 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 librados: starting msgr at 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 librados: starting objecter 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 Processor -- start 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 -- start start 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 librados: setting wanted keys 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 1 librados: calling monclient init 2024-03-25T18:14:09.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.518+0000 7f6a13715640 10 monclient: init 2024-03-25T18:14:09.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:09.521 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c070240 con 0x7f6a0c1430c0 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c071b30 con 0x7f6a0c13f710 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c072020 con 0x7f6a0c0a98f0 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.519+0000 7f6a13715640 10 monclient(hunting): authenticate will time out at 7558.781238s 2024-03-25T18:14:09.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.527+0000 7f6a10cb0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6a0c13f710 0x7f6a0c138fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:36800/0 (socket says 172.21.15.46:36800) 2024-03-25T18:14:09.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.527+0000 7f6a10cb0640 1 -- 172.21.15.46:0/918130498 learned_addr learned my addr 172.21.15.46:0/918130498 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:09.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.529+0000 7f6a01ffb640 10 client.?.objecter ms_handle_connect 0x7f6a0c13f710 2024-03-25T18:14:09.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.529+0000 7f6a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.539+0000 7f6a01ffb640 10 client.?.objecter ms_handle_connect 0x7f6a0c0a98f0 2024-03-25T18:14:09.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.539+0000 7f6a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195335689 0 0) 0x7f6a0c071b30 con 0x7f6a0c13f710 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:14:09.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.541+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f0004eb0 con 0x7f6a0c13f710 2024-03-25T18:14:09.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.543+0000 7f6a01ffb640 10 client.?.objecter ms_handle_connect 0x7f6a0c1430c0 2024-03-25T18:14:09.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.543+0000 7f6a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577181578 0 0) 0x7f6a0c072020 con 0x7f6a0c0a98f0 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:14:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.553+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c071b30 con 0x7f6a0c0a98f0 2024-03-25T18:14:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721853582 0 0) 0x7f6a0c070240 con 0x7f6a0c1430c0 2024-03-25T18:14:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:14:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.556+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c072020 con 0x7f6a0c1430c0 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293964072 0 0) 0x7f69f0004eb0 con 0x7f6a0c13f710 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a0c070240 con 0x7f6a0c13f710 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f69fc001a30 con 0x7f6a0c13f710 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.558 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.557+0000 7f6a01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3144364055 0 0) 0x7f6a0c070240 con 0x7f6a0c13f710 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 >> v1:172.21.15.46:6790/0 conn(0x7f6a0c0a98f0 legacy=0x7f6a0c06ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 >> v1:172.21.15.46:6789/0 conn(0x7f6a0c1430c0 legacy=0x7f6a0c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 1 monclient: found mon.b 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c0704b0 con 0x7f6a0c13f710 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a13715640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a13715640 10 monclient: _renew_subs 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a13715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.561+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0c06f120 con 0x7f6a0c13f710 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.562+0000 7f6a13715640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.562+0000 7f6a13715640 10 monclient: _renew_subs 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.562+0000 7f6a13715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.562+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0c1396e0 con 0x7f6a0c13f710 2024-03-25T18:14:09.564 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.562+0000 7f6a13715640 1 librados: init done 2024-03-25T18:14:09.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.563+0000 7f6a13715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.563+0000 7f6a13715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.563+0000 7f6a13715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:09.565 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.563+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0c0627b0 con 0x7f6a0c13f710 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69fc001c70 con 0x7f6a0c13f710 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f69fc003a60 con 0x7f6a0c13f710 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a01ffb640 10 monclient: dump: 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.839 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.838+0000 7f6a037fe640 4 set_mon_vals no callback set 2024-03-25T18:14:09.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.839+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f69fc003c40 con 0x7f6a0c13f710 2024-03-25T18:14:09.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.839+0000 7f6a114b1640 10 monclient: get_auth_request con 0x7f69f006ddf0 auth_method 0 2024-03-25T18:14:09.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.840+0000 7f6a01ffb640 10 client.4468.objecter ms_handle_connect 0x7f69f006ddf0 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f69fc054900 con 0x7f6a0c13f710 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 10 client.4468.objecter ms_dispatch 0x7f6a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69fc050240 con 0x7f6a0c13f710 2024-03-25T18:14:09.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:09.844+0000 7f6a01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:10.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.102+0000 7f6a13715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.102+0000 7f6a13715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.102+0000 7f6a13715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:10.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.102+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a0c1399e0 con 0x7f6a0c13f710 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.127+0000 7f6a01ffb640 1 -- 172.21.15.46:0/918130498 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f69fc052340 con 0x7f6a0c13f710 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.127+0000 7f6a01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.128 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.127+0000 7f6a01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:10.128 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:10.128 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 10 librados: watch_flush enter 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 10 librados: watch_flush exit 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 10 client.4468.objecter successfully canceled tick 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 >> v1:172.21.15.164:6800/1612084612 conn(0x7f69f006ddf0 legacy=0x7f69f00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 10 monclient: shutdown 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 >> v1:172.21.15.164:6789/0 conn(0x7f6a0c13f710 legacy=0x7f6a0c138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 shutdown_connections 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 >> 172.21.15.46:0/918130498 conn(0x7f6a0c0b7ea0 msgr2=0x7f6a0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 shutdown_connections 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 -- 172.21.15.46:0/918130498 wait complete. 2024-03-25T18:14:10.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:10.129+0000 7f6a13715640 1 librados: shutdown 2024-03-25T18:14:10.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:16.949 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:17.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 10 monclient: monmap: 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:17.065777+0000 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:17.065777+0000 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:17.066 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:17.067 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 1 Processor -- start 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 1 -- start start 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.065+0000 7fe45b7f6640 10 monclient: init 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4540a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe45413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4541430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe454142520 con 0x7fe4541430c0 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4540b28a0 con 0x7fe4540a98f0 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4540b2a30 con 0x7fe45413f710 2024-03-25T18:14:17.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.066+0000 7fe45b7f6640 10 monclient(hunting): authenticate will time out at 7566.328532s 2024-03-25T18:14:17.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.081+0000 7fe459d93640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4541430c0 0x7fe45414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39650/0 (socket says 172.21.15.46:39650) 2024-03-25T18:14:17.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.081+0000 7fe459d93640 1 -- 172.21.15.46:0/1490261684 learned_addr learned my addr 172.21.15.46:0/1490261684 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346658030 0 0) 0x7fe454142520 con 0x7fe4541430c0 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:14:17.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.098+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe440003520 con 0x7fe4541430c0 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666556167 0 0) 0x7fe4540b28a0 con 0x7fe4540a98f0 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:14:17.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.102+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe454142520 con 0x7fe4540a98f0 2024-03-25T18:14:17.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934597513 0 0) 0x7fe4540b2a30 con 0x7fe45413f710 2024-03-25T18:14:17.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.105 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:14:17.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.104+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4540b28a0 con 0x7fe45413f710 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941256957 0 0) 0x7fe454142520 con 0x7fe4540a98f0 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 >> v1:172.21.15.46:6790/0 conn(0x7fe45413f710 legacy=0x7fe4541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 >> v1:172.21.15.46:6789/0 conn(0x7fe4541430c0 legacy=0x7fe45414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 monclient: found mon.noname-c 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4540b44c0 con 0x7fe4540a98f0 2024-03-25T18:14:17.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe450001670 con 0x7fe4540a98f0 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe45b7f6640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe45b7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe44bfff640 10 monclient: dump: 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.119 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:17.120 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:17.120 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:17.120 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.117+0000 7fe45b7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:17.151 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4500017d0 con 0x7fe4540a98f0 2024-03-25T18:14:17.151 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 1 -- 172.21.15.46:0/1490261684 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe450001be0 con 0x7fe4540a98f0 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe44bfff640 10 monclient: dump: 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe45b7f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:17.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.151+0000 7fe45b7f6640 4 set_mon_vals no callback set 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 10 monclient: shutdown 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1490261684 >> v1:172.21.15.164:6789/0 conn(0x7fe4540a98f0 legacy=0x7fe4540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1490261684 shutdown_connections 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1490261684 >> 172.21.15.46:0/1490261684 conn(0x7fe4540b7ea0 msgr2=0x7fe4540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1490261684 shutdown_connections 2024-03-25T18:14:17.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.152+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1490261684 wait complete. 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 10 monclient: monmap: 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 librados: starting msgr at 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 librados: starting objecter 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 Processor -- start 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 -- start start 2024-03-25T18:14:17.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 librados: setting wanted keys 2024-03-25T18:14:17.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 1 librados: calling monclient init 2024-03-25T18:14:17.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.153+0000 7fe45b7f6640 10 monclient: init 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4540a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe45413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4541430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4541390f0 con 0x7fe45413f710 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe454138dd0 con 0x7fe4541430c0 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4541346b0 con 0x7fe4540a98f0 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.154+0000 7fe45b7f6640 10 monclient(hunting): authenticate will time out at 7566.416087s 2024-03-25T18:14:17.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.164+0000 7fe459592640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fe4540a98f0 0x7fe454137e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:38720/0 (socket says 172.21.15.46:38720) 2024-03-25T18:14:17.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.164+0000 7fe459592640 1 -- 172.21.15.46:0/1346293636 learned_addr learned my addr 172.21.15.46:0/1346293636 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:17.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.165+0000 7fe449ffb640 10 client.?.objecter ms_handle_connect 0x7fe4541430c0 2024-03-25T18:14:17.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.165+0000 7fe449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.168+0000 7fe449ffb640 10 client.?.objecter ms_handle_connect 0x7fe4540a98f0 2024-03-25T18:14:17.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.168+0000 7fe449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903704892 0 0) 0x7fe4541346b0 con 0x7fe4540a98f0 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:14:17.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.176+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe438004eb0 con 0x7fe4540a98f0 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803520948 0 0) 0x7fe454138dd0 con 0x7fe4541430c0 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4541346b0 con 0x7fe4541430c0 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 client.?.objecter ms_handle_connect 0x7fe45413f710 2024-03-25T18:14:17.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.177+0000 7fe449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2124699760 0 0) 0x7fe4541346b0 con 0x7fe4541430c0 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe454138dd0 con 0x7fe4541430c0 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe44c001af0 con 0x7fe4541430c0 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:17.190 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:17.191 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:17.191 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.189+0000 7fe449ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206759310 0 0) 0x7fe438004eb0 con 0x7fe4540a98f0 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4541346b0 con 0x7fe4540a98f0 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe450001670 con 0x7fe4540a98f0 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.199+0000 7fe449ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.200 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:17.201 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399708093 0 0) 0x7fe454138dd0 con 0x7fe4541430c0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 >> v1:172.21.15.46:6790/0 conn(0x7fe4540a98f0 legacy=0x7fe454137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 >> v1:172.21.15.46:6789/0 conn(0x7fe45413f710 legacy=0x7fe4541386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 1 monclient: found mon.b 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe454134840 con 0x7fe4541430c0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe45b7f6640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe45b7f6640 10 monclient: _renew_subs 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe45b7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.200+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe454134a90 con 0x7fe4541430c0 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.201+0000 7fe45b7f6640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.201+0000 7fe45b7f6640 10 monclient: _renew_subs 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.201+0000 7fe45b7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.201+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4541356f0 con 0x7fe4541430c0 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.201+0000 7fe45b7f6640 1 librados: init done 2024-03-25T18:14:17.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.202+0000 7fe45b7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.202+0000 7fe45b7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.202+0000 7fe45b7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.202+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4540627b0 con 0x7fe4541430c0 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe44c001f00 con 0x7fe4541430c0 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe44c004b60 con 0x7fe4541430c0 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe449ffb640 10 monclient: dump: 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.473 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:17.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.472+0000 7fe44b7fe640 4 set_mon_vals no callback set 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.473+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fe44c004d80 con 0x7fe4541430c0 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.473+0000 7fe459592640 10 monclient: get_auth_request con 0x7fe43806e030 auth_method 0 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.474+0000 7fe449ffb640 10 client.4474.objecter ms_handle_connect 0x7fe43806e030 2024-03-25T18:14:17.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fe44c054910 con 0x7fe4541430c0 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 10 client.4474.objecter ms_dispatch 0x7fe454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe44c050250 con 0x7fe4541430c0 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.481+0000 7fe449ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:17.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.737+0000 7fe45b7f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.737+0000 7fe45b7f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.737+0000 7fe45b7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:17.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.737+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4541361b0 con 0x7fe4541430c0 2024-03-25T18:14:17.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.764+0000 7fe449ffb640 1 -- 172.21.15.46:0/1346293636 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fe44c052350 con 0x7fe4541430c0 2024-03-25T18:14:17.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.764+0000 7fe449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.764+0000 7fe449ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:17.765 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:17.765 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:17.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 10 librados: watch_flush enter 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 10 librados: watch_flush exit 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 >> v1:172.21.15.164:6800/1612084612 conn(0x7fe43806e030 legacy=0x7fe4380704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 10 monclient: shutdown 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 >> v1:172.21.15.164:6789/0 conn(0x7fe4541430c0 legacy=0x7fe454133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 shutdown_connections 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 >> 172.21.15.46:0/1346293636 conn(0x7fe4540b7ea0 msgr2=0x7fe454074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 shutdown_connections 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 -- 172.21.15.46:0/1346293636 wait complete. 2024-03-25T18:14:17.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:17.766+0000 7fe45b7f6640 1 librados: shutdown 2024-03-25T18:14:17.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:19.718 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:55.636 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.765 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:55.682 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.811 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:05:55.729 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.817 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:55.688 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.822 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:55.693 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.837 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:55.754 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.842 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:55.760 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.851 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:05:55.722 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.889 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:05:55.806 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:24.586 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:24.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.699+0000 7ff7fa4fb640 10 monclient: get_monmap_and_config 2024-03-25T18:14:24.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.699+0000 7ff7fa4fb640 10 monclient: build_initial_monmap 2024-03-25T18:14:24.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.699+0000 7ff7fa4fb640 10 monclient: monmap: 2024-03-25T18:14:24.700 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:24.700 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:24.700465+0000 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:24.700465+0000 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:24.701 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.700+0000 7ff7fa4fb640 1 Processor -- start 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 1 -- start start 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient: init 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f40a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4142520 con 0x7ff7f413f710 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f40b28a0 con 0x7ff7f40a98f0 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f40b2a30 con 0x7ff7f41430c0 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:24.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.701+0000 7ff7fa4fb640 10 monclient(hunting): authenticate will time out at 7573.963432s 2024-03-25T18:14:24.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.706+0000 7ff7f37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff7f413f710 0x7ff7f41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36030/0 (socket says 172.21.15.46:36030) 2024-03-25T18:14:24.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.706+0000 7ff7f37fe640 1 -- 172.21.15.46:0/119384974 learned_addr learned my addr 172.21.15.46:0/119384974 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:24.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140301536 0 0) 0x7ff7f4142520 con 0x7ff7f413f710 2024-03-25T18:14:24.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:14:24.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.726+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003520 con 0x7ff7f413f710 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847073691 0 0) 0x7ff7f40b2a30 con 0x7ff7f41430c0 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:14:24.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.732+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f4142520 con 0x7ff7f41430c0 2024-03-25T18:14:24.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470882215 0 0) 0x7ff7e0003520 con 0x7ff7f413f710 2024-03-25T18:14:24.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 >> v1:172.21.15.46:6790/0 conn(0x7ff7f41430c0 legacy=0x7ff7f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 >> v1:172.21.15.164:6789/0 conn(0x7ff7f40a98f0 legacy=0x7ff7f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f40b44c0 con 0x7ff7f413f710 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7fa4fb640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7fa4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff7e4001690 con 0x7ff7f413f710 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7f2ffd640 10 monclient: dump: 2024-03-25T18:14:24.737 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.735+0000 7ff7fa4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7e40018e0 con 0x7ff7f413f710 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 1 -- 172.21.15.46:0/119384974 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff7e4001c70 con 0x7ff7f413f710 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7fa4fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7fa4fb640 4 set_mon_vals no callback set 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.752+0000 7ff7f2ffd640 10 monclient: dump: 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.753 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.754 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:24.754 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:24.754 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:24.754 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:24.754 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 10 monclient: shutdown 2024-03-25T18:14:24.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:24.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/119384974 >> v1:172.21.15.46:6789/0 conn(0x7ff7f413f710 legacy=0x7ff7f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/119384974 shutdown_connections 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/119384974 >> 172.21.15.46:0/119384974 conn(0x7ff7f40b7ea0 msgr2=0x7ff7f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/119384974 shutdown_connections 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.753+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/119384974 wait complete. 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.754+0000 7ff7fa4fb640 10 monclient: build_initial_monmap 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.754+0000 7ff7fa4fb640 10 monclient: monmap: 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:24.756 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.754+0000 7ff7fa4fb640 1 librados: starting msgr at 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.754+0000 7ff7fa4fb640 1 librados: starting objecter 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.754+0000 7ff7fa4fb640 1 Processor -- start 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 -- start start 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 librados: setting wanted keys 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 librados: calling monclient init 2024-03-25T18:14:24.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient: init 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f40a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f41430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4135970 con 0x7ff7f413f710 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4136e90 con 0x7ff7f41430c0 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f4137f10 con 0x7ff7f40a98f0 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:24.758 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.755+0000 7ff7fa4fb640 10 monclient(hunting): authenticate will time out at 7574.017321s 2024-03-25T18:14:24.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.758+0000 7ff7f37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff7f413f710 0x7ff7f41db2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36072/0 (socket says 172.21.15.46:36072) 2024-03-25T18:14:24.759 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.758+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1847973167 learned_addr learned my addr 172.21.15.46:0/1847973167 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:24.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.763+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f413f710 2024-03-25T18:14:24.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.763+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.767+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688562389 0 0) 0x7ff7f4135970 con 0x7ff7f413f710 2024-03-25T18:14:24.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.768+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.768+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.768+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.769 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.768+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:14:24.769 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.768+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d4004eb0 con 0x7ff7f413f710 2024-03-25T18:14:24.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.772+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f41430c0 2024-03-25T18:14:24.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.772+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.776+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1287204951 0 0) 0x7ff7d4004eb0 con 0x7ff7f413f710 2024-03-25T18:14:24.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.776+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f4135970 con 0x7ff7f413f710 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.778+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff7e4001c90 con 0x7ff7f413f710 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.778+0000 7ff7f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.778+0000 7ff7f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.778+0000 7ff7f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:24.779 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.778+0000 7ff7f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272608591 0 0) 0x7ff7f4136e90 con 0x7ff7f41430c0 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:14:24.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.779+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d4004eb0 con 0x7ff7f41430c0 2024-03-25T18:14:24.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.783+0000 7ff7f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff7f40a98f0 2024-03-25T18:14:24.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.783+0000 7ff7f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312608776 0 0) 0x7ff7d4004eb0 con 0x7ff7f41430c0 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f4136e90 con 0x7ff7f41430c0 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff7dc0016b0 con 0x7ff7f41430c0 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:24.793 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915837261 0 0) 0x7ff7f4137f10 con 0x7ff7f40a98f0 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:14:24.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.792+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d4004eb0 con 0x7ff7f40a98f0 2024-03-25T18:14:24.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1877284085 0 0) 0x7ff7f4136e90 con 0x7ff7f41430c0 2024-03-25T18:14:24.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 >> v1:172.21.15.46:6790/0 conn(0x7ff7f40a98f0 legacy=0x7ff7f4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 >> v1:172.21.15.46:6789/0 conn(0x7ff7f413f710 legacy=0x7ff7f41db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 1 monclient: found mon.b 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f41dc0e0 con 0x7ff7f41430c0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 10 monclient: _renew_subs 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f4137100 con 0x7ff7f41430c0 2024-03-25T18:14:24.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 10 monclient: _renew_subs 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f40b7bb0 con 0x7ff7f41430c0 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.808+0000 7ff7fa4fb640 1 librados: init done 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.809+0000 7ff7fa4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.809+0000 7ff7fa4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:24.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.809+0000 7ff7fa4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:24.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:24.809+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f40627b0 con 0x7ff7f41430c0 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7dc001ba0 con 0x7ff7f41430c0 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff7dc0037b0 con 0x7ff7f41430c0 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f0ff9640 10 monclient: dump: 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:25.077 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:25.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.076+0000 7ff7f27fc640 4 set_mon_vals no callback set 2024-03-25T18:14:25.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.077+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7ff7dc019780 con 0x7ff7f41430c0 2024-03-25T18:14:25.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.078+0000 7ff7f3fff640 10 monclient: get_auth_request con 0x7ff7d406df00 auth_method 0 2024-03-25T18:14:25.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.078+0000 7ff7f0ff9640 10 client.4477.objecter ms_handle_connect 0x7ff7d406df00 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7ff7dc0533d0 con 0x7ff7f41430c0 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 10 client.4477.objecter ms_dispatch 0x7ff7f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:14:25.082 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7dc04eee0 con 0x7ff7f41430c0 2024-03-25T18:14:25.083 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.083 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.081+0000 7ff7f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:25.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.333+0000 7ff7fa4fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.333+0000 7ff7fa4fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.333+0000 7ff7fa4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:25.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.333+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7f413e250 con 0x7ff7f41430c0 2024-03-25T18:14:25.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.353+0000 7ff7f0ff9640 1 -- 172.21.15.46:0/1847973167 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7ff7dc050e10 con 0x7ff7f41430c0 2024-03-25T18:14:25.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.353+0000 7ff7f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.353+0000 7ff7f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:25.354 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:25.354 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:25.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 10 librados: watch_flush enter 2024-03-25T18:14:25.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 10 librados: watch_flush exit 2024-03-25T18:14:25.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:14:25.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:14:25.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 >> v1:172.21.15.164:6800/1612084612 conn(0x7ff7d406df00 legacy=0x7ff7d40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 10 monclient: shutdown 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 >> v1:172.21.15.164:6789/0 conn(0x7ff7f41430c0 legacy=0x7ff7f41db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 shutdown_connections 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 >> 172.21.15.46:0/1847973167 conn(0x7ff7f40b7ea0 msgr2=0x7ff7f414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 shutdown_connections 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 -- 172.21.15.46:0/1847973167 wait complete. 2024-03-25T18:14:25.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:25.356+0000 7ff7fa4fb640 1 librados: shutdown 2024-03-25T18:14:25.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:32.176 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:32.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.292+0000 7fcb20dcf640 10 monclient: get_monmap_and_config 2024-03-25T18:14:32.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.292+0000 7fcb20dcf640 10 monclient: build_initial_monmap 2024-03-25T18:14:32.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.292+0000 7fcb20dcf640 10 monclient: monmap: 2024-03-25T18:14:32.293 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:32.293 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:32.293524+0000 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:32.293524+0000 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:32.294 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.293+0000 7fcb20dcf640 1 Processor -- start 2024-03-25T18:14:32.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.293+0000 7fcb20dcf640 1 -- start start 2024-03-25T18:14:32.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.293+0000 7fcb20dcf640 10 monclient: init 2024-03-25T18:14:32.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fcb1c0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fcb1c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fcb1c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c142520 con 0x7fcb1c1430c0 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0b28a0 con 0x7fcb1c13f710 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c0b2a30 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:32.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.294+0000 7fcb20dcf640 10 monclient(hunting): authenticate will time out at 7581.556201s 2024-03-25T18:14:32.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.306+0000 7fcb1ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcb1c1430c0 0x7fcb1c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40602/0 (socket says 172.21.15.46:40602) 2024-03-25T18:14:32.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.306+0000 7fcb1ad9d640 1 -- 172.21.15.46:0/3846712444 learned_addr learned my addr 172.21.15.46:0/3846712444 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466550195 0 0) 0x7fcb1c0b2a30 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:14:32.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.318+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08003520 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787386064 0 0) 0x7fcb08003520 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 >> v1:172.21.15.164:6789/0 conn(0x7fcb1c13f710 legacy=0x7fcb1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 >> v1:172.21.15.46:6789/0 conn(0x7fcb1c1430c0 legacy=0x7fcb1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 monclient: found mon.noname-b 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:32.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0b44c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: _finish_auth 0 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb0c001690 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb1959a640 10 monclient: dump: 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.330 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb20dcf640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:14:32.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.328+0000 7fcb20dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb0c0017f0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 1 -- 172.21.15.46:0/3846712444 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb0c001c00 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb20dcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb20dcf640 4 set_mon_vals no callback set 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.337+0000 7fcb1959a640 10 monclient: dump: 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:32.339 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 10 monclient: shutdown 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3846712444 >> v1:172.21.15.46:6790/0 conn(0x7fcb1c0a98f0 legacy=0x7fcb1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3846712444 shutdown_connections 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3846712444 >> 172.21.15.46:0/3846712444 conn(0x7fcb1c0b7ea0 msgr2=0x7fcb1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3846712444 shutdown_connections 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.338+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3846712444 wait complete. 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.339+0000 7fcb20dcf640 10 monclient: build_initial_monmap 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.339+0000 7fcb20dcf640 10 monclient: monmap: 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:32.341 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 librados: starting msgr at 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 librados: starting objecter 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 Processor -- start 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 -- start start 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 librados: setting wanted keys 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 1 librados: calling monclient init 2024-03-25T18:14:32.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.340+0000 7fcb20dcf640 10 monclient: init 2024-03-25T18:14:32.343 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:32.343 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fcb1c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fcb1c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fcb1c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c135730 con 0x7fcb1c1430c0 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c1358c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c135a50 con 0x7fcb1c13f710 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:32.344 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.341+0000 7fcb20dcf640 10 monclient(hunting): authenticate will time out at 7581.602755s 2024-03-25T18:14:32.349 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.348+0000 7fcb1a59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcb1c0a98f0 0x7fcb1c1dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:43558/0 (socket says 172.21.15.46:43558) 2024-03-25T18:14:32.349 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.348+0000 7fcb1a59c640 1 -- 172.21.15.46:0/3065080849 learned_addr learned my addr 172.21.15.46:0/3065080849 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:32.352 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.351+0000 7fcb12ffd640 10 client.?.objecter ms_handle_connect 0x7fcb1c0a98f0 2024-03-25T18:14:32.352 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.351+0000 7fcb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.359 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.359+0000 7fcb12ffd640 10 client.?.objecter ms_handle_connect 0x7fcb1c1430c0 2024-03-25T18:14:32.359 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.359+0000 7fcb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120163334 0 0) 0x7fcb1c1358c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:14:32.362 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.361+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcafc004eb0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.363+0000 7fcb12ffd640 10 client.?.objecter ms_handle_connect 0x7fcb1c13f710 2024-03-25T18:14:32.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.363+0000 7fcb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956666402 0 0) 0x7fcafc004eb0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb1c1358c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb0c001bc0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.369 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.370 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.368+0000 7fcb12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810575977 0 0) 0x7fcb1c135a50 con 0x7fcb1c13f710 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:14:32.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.374+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcafc004eb0 con 0x7fcb1c13f710 2024-03-25T18:14:32.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120522385 0 0) 0x7fcb1c135730 con 0x7fcb1c1430c0 2024-03-25T18:14:32.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.380 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:14:32.380 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.379+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb1c135a50 con 0x7fcb1c1430c0 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 247300453 0 0) 0x7fcb1c1358c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 >> v1:172.21.15.46:6790/0 conn(0x7fcb1c13f710 legacy=0x7fcb1c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 >> v1:172.21.15.46:6789/0 conn(0x7fcb1c1430c0 legacy=0x7fcb1c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 1 monclient: found mon.b 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c1d5000 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 10 monclient: _renew_subs 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c1d52b0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 10 monclient: _renew_subs 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c13c900 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.383+0000 7fcb20dcf640 1 librados: init done 2024-03-25T18:14:32.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.384+0000 7fcb20dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.384+0000 7fcb20dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.384+0000 7fcb20dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:32.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.384+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c1371c0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb0c0011d0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fcb0c003b00 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb12ffd640 10 monclient: dump: 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:32.411 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.410+0000 7fcb18d99640 4 set_mon_vals no callback set 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.411+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fcb0c003d20 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.412+0000 7fcb19d9b640 10 monclient: get_auth_request con 0x7fcafc06de80 auth_method 0 2024-03-25T18:14:32.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.412+0000 7fcb12ffd640 10 client.4483.objecter ms_handle_connect 0x7fcafc06de80 2024-03-25T18:14:32.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.414+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fcb0c0548d0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.414+0000 7fcb12ffd640 10 client.4483.objecter ms_dispatch 0x7fcb1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:32.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.414+0000 7fcb12ffd640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:32.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.414+0000 7fcb12ffd640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:32.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.414+0000 7fcb12ffd640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:14:32.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.664+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb0c0503e0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.664+0000 7fcb12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.665 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.664+0000 7fcb12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:32.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.896+0000 7fcb20dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.896+0000 7fcb20dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.896+0000 7fcb20dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:32.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.896+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb1c0627b0 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.924 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.924+0000 7fcb12ffd640 1 -- 172.21.15.46:0/3065080849 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fcb0c052310 con 0x7fcb1c0a98f0 2024-03-25T18:14:32.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.924+0000 7fcb12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.925 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.924+0000 7fcb12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:32.925 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:32.925 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:32.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 10 librados: watch_flush enter 2024-03-25T18:14:32.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 10 librados: watch_flush exit 2024-03-25T18:14:32.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 >> v1:172.21.15.164:6800/1612084612 conn(0x7fcafc06de80 legacy=0x7fcafc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 10 monclient: shutdown 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 >> v1:172.21.15.164:6789/0 conn(0x7fcb1c0a98f0 legacy=0x7fcb1c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 shutdown_connections 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 >> 172.21.15.46:0/3065080849 conn(0x7fcb1c0b7ea0 msgr2=0x7fcb1c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 shutdown_connections 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 -- 172.21.15.46:0/3065080849 wait complete. 2024-03-25T18:14:32.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:32.926+0000 7fcb20dcf640 1 librados: shutdown 2024-03-25T18:14:32.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:39.745 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:39.768 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:15.686 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.822 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:15.740 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.839+0000 7f59ccf23640 10 monclient: get_monmap_and_config 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.839+0000 7f59ccf23640 10 monclient: build_initial_monmap 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.839+0000 7f59ccf23640 10 monclient: monmap: 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:39.840363+0000 2024-03-25T18:14:39.840 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:39.840363+0000 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:39.841 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.840+0000 7f59ccf23640 1 Processor -- start 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.840+0000 7f59ccf23640 1 -- start start 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.840+0000 7f59ccf23640 10 monclient: init 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-c con 0x7f59c80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:39.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-a con 0x7f59c813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-b con 0x7f59c81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c8142520 con 0x7f59c813f710 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80b28a0 con 0x7f59c80a98f0 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80b2a30 con 0x7f59c81430c0 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:39.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.841+0000 7f59ccf23640 10 monclient(hunting): authenticate will time out at 7589.102709s 2024-03-25T18:14:39.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.864+0000 7f59c6d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f59c81430c0 0x7f59c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:59196/0 (socket says 172.21.15.46:59196) 2024-03-25T18:14:39.864 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.864+0000 7f59c6d9d640 1 -- 172.21.15.46:0/1898534915 learned_addr learned my addr 172.21.15.46:0/1898534915 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:39.872 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:15.789 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.876 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:15.747 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.882 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:15.753 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.888 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:15.806 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.895 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:15.813 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217052328 0 0) 0x7f59c80b28a0 con 0x7f59c80a98f0 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b0003520 con 0x7f59c80a98f0 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735367471 0 0) 0x7f59c8142520 con 0x7f59c813f710 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:14:39.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.911+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c80b28a0 con 0x7f59c813f710 2024-03-25T18:14:39.920 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:15.791 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299857819 0 0) 0x7f59c80b28a0 con 0x7f59c813f710 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 >> v1:172.21.15.46:6790/0 conn(0x7f59c81430c0 legacy=0x7f59c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 >> v1:172.21.15.164:6789/0 conn(0x7f59c80a98f0 legacy=0x7f59c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 monclient: found mon.noname-a 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:14:39.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80b44c0 con 0x7f59c813f710 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f59bc001670 con 0x7f59c813f710 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59ccf23640 5 monclient: authenticate success, global_id 4488 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59ccf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59c559a640 10 monclient: dump: 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:39.930 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:39.931 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:39.931 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:39.931 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:39.931 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.928+0000 7f59ccf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:39.931 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:15.847 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.948 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59bc0017d0 con 0x7f59c813f710 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 1 -- 172.21.15.46:0/1898534915 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f59bc001ba0 con 0x7f59c813f710 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59c559a640 10 monclient: dump: 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:39.949 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:39.950 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:39.950 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:39.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59ccf23640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:39.950 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.948+0000 7f59ccf23640 4 set_mon_vals no callback set 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 10 monclient: shutdown 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 1 -- 172.21.15.46:0/1898534915 >> v1:172.21.15.46:6789/0 conn(0x7f59c813f710 legacy=0x7f59c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 1 -- 172.21.15.46:0/1898534915 shutdown_connections 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 1 -- 172.21.15.46:0/1898534915 >> 172.21.15.46:0/1898534915 conn(0x7f59c80b7ea0 msgr2=0x7f59c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 1 -- 172.21.15.46:0/1898534915 shutdown_connections 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.949+0000 7f59ccf23640 1 -- 172.21.15.46:0/1898534915 wait complete. 2024-03-25T18:14:39.951 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 10 monclient: build_initial_monmap 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 10 monclient: monmap: 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 librados: starting msgr at 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 librados: starting objecter 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 Processor -- start 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 -- start start 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 librados: setting wanted keys 2024-03-25T18:14:39.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 1 librados: calling monclient init 2024-03-25T18:14:39.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.950+0000 7f59ccf23640 10 monclient: init 2024-03-25T18:14:39.953 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-a con 0x7f59c80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-c con 0x7f59c813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient(hunting): picked mon.noname-b con 0x7f59c81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c81d7830 con 0x7f59c80a98f0 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c81d7500 con 0x7f59c81430c0 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c8137970 con 0x7f59c813f710 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:39.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.951+0000 7f59ccf23640 10 monclient(hunting): authenticate will time out at 7589.212812s 2024-03-25T18:14:39.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.997+0000 7f59c6d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f59c81430c0 0x7f59c81e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:55232/0 (socket says 172.21.15.46:55232) 2024-03-25T18:14:39.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:39.997+0000 7f59c6d9d640 1 -- 172.21.15.46:0/4154646213 learned_addr learned my addr 172.21.15.46:0/4154646213 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:40.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.000+0000 7f59b6ffd640 10 client.?.objecter ms_handle_connect 0x7f59c81430c0 2024-03-25T18:14:40.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.000+0000 7f59b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.007+0000 7f59b6ffd640 10 client.?.objecter ms_handle_connect 0x7f59c80a98f0 2024-03-25T18:14:40.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.007+0000 7f59b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.010+0000 7f59b6ffd640 10 client.?.objecter ms_handle_connect 0x7f59c813f710 2024-03-25T18:14:40.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.010+0000 7f59b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980301098 0 0) 0x7f59c8137970 con 0x7f59c813f710 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:14:40.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.030+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a8004eb0 con 0x7f59c813f710 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155425256 0 0) 0x7f59c81d7500 con 0x7f59c81430c0 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:14:40.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.033+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c8137970 con 0x7f59c81430c0 2024-03-25T18:14:40.035 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653429687 0 0) 0x7f59c81d7830 con 0x7f59c80a98f0 2024-03-25T18:14:40.035 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.035 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:14:40.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.035+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c81d7500 con 0x7f59c80a98f0 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3985377669 0 0) 0x7f59a8004eb0 con 0x7f59c813f710 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59c81d7830 con 0x7f59c813f710 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f59bc001ae0 con 0x7f59c813f710 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.075+0000 7f59b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.077 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:40.078 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:40.078 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:40.078 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:40.078 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:40.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.076+0000 7f59b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:40.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 175329543 0 0) 0x7f59c81d7830 con 0x7f59c813f710 2024-03-25T18:14:40.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 >> v1:172.21.15.164:6789/0 conn(0x7f59c81430c0 legacy=0x7f59c81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 >> v1:172.21.15.46:6789/0 conn(0x7f59c80a98f0 legacy=0x7f59c8138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 1 monclient: found mon.c 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8137b00 con 0x7f59c813f710 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 5 monclient: authenticate success, global_id 4502 2024-03-25T18:14:40.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 10 monclient: _renew_subs 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c8135080 con 0x7f59c813f710 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 10 monclient: _renew_subs 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81e36d0 con 0x7f59c813f710 2024-03-25T18:14:40.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.092+0000 7f59ccf23640 1 librados: init done 2024-03-25T18:14:40.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.094+0000 7f59ccf23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.094+0000 7f59ccf23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.094+0000 7f59ccf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:40.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.094+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c81d7690 con 0x7f59c813f710 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59bc0038a0 con 0x7f59c813f710 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f59bc003a10 con 0x7f59c813f710 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59b6ffd640 10 monclient: dump: 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:40.407 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:40.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.406+0000 7f59c4d99640 4 set_mon_vals no callback set 2024-03-25T18:14:40.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.407+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f59bc0199e0 con 0x7f59c813f710 2024-03-25T18:14:40.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.408+0000 7f59c659c640 10 monclient: get_auth_request con 0x7f59a806ddf0 auth_method 0 2024-03-25T18:14:40.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.408+0000 7f59b6ffd640 10 client.4502.objecter ms_handle_connect 0x7f59a806ddf0 2024-03-25T18:14:40.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f59bc053590 con 0x7f59c813f710 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 10 client.4502.objecter ms_dispatch 0x7f59c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59bc04f0a0 con 0x7f59c813f710 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.412+0000 7f59b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:40.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.673+0000 7f59ccf23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.673+0000 7f59ccf23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.673+0000 7f59ccf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:14:40.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.673+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c80627b0 con 0x7f59c813f710 2024-03-25T18:14:40.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.697+0000 7f59b6ffd640 1 -- 172.21.15.46:0/4154646213 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f59bc050fd0 con 0x7f59c813f710 2024-03-25T18:14:40.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.697+0000 7f59b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.697+0000 7f59b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:40.699 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:40.699 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:40.701 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 10 librados: watch_flush enter 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 10 librados: watch_flush exit 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 10 client.4502.objecter successfully canceled tick 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 >> v1:172.21.15.164:6800/1612084612 conn(0x7f59a806ddf0 legacy=0x7f59a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 10 monclient: shutdown 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 >> v1:172.21.15.46:6790/0 conn(0x7f59c813f710 legacy=0x7f59c81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 shutdown_connections 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 >> 172.21.15.46:0/4154646213 conn(0x7f59c80b7ea0 msgr2=0x7f59c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 shutdown_connections 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.700+0000 7f59ccf23640 1 -- 172.21.15.46:0/4154646213 wait complete. 2024-03-25T18:14:40.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:40.701+0000 7f59ccf23640 1 librados: shutdown 2024-03-25T18:14:40.714 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:47.521 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:47.636 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.636+0000 7fd9bcea8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.636+0000 7fd9bcea8640 10 monclient: build_initial_monmap 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.636+0000 7fd9bcea8640 10 monclient: monmap: 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:47.636995+0000 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:47.636995+0000 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:47.637 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 1 Processor -- start 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 1 -- start start 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient: init 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b80a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8142520 con 0x7fd9b81430c0 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b80b28a0 con 0x7fd9b813f710 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b80b2a30 con 0x7fd9b80a98f0 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:47.639 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.637+0000 7fd9bcea8640 10 monclient(hunting): authenticate will time out at 7596.899508s 2024-03-25T18:14:47.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.648+0000 7fd9b6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd9b81430c0 0x7fd9b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36926/0 (socket says 172.21.15.46:36926) 2024-03-25T18:14:47.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.648+0000 7fd9b6d9d640 1 -- 172.21.15.46:0/1836948115 learned_addr learned my addr 172.21.15.46:0/1836948115 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:47.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893823555 0 0) 0x7fd9b8142520 con 0x7fd9b81430c0 2024-03-25T18:14:47.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.661 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:14:47.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.660+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a0003520 con 0x7fd9b81430c0 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501189506 0 0) 0x7fd9b80b28a0 con 0x7fd9b813f710 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:14:47.670 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.669+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8142520 con 0x7fd9b813f710 2024-03-25T18:14:47.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362120802 0 0) 0x7fd9a0003520 con 0x7fd9b81430c0 2024-03-25T18:14:47.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 >> v1:172.21.15.46:6790/0 conn(0x7fd9b80a98f0 legacy=0x7fd9b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 >> v1:172.21.15.164:6789/0 conn(0x7fd9b813f710 legacy=0x7fd9b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:47.673 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 monclient: found mon.noname-a 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b80b44c0 con 0x7fd9b81430c0 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9b0001a10 con 0x7fd9b81430c0 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9bcea8640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9bcea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9b559a640 10 monclient: dump: 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.674 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.675 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.672+0000 7fd9bcea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9b0001b70 con 0x7fd9b81430c0 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 1 -- 172.21.15.46:0/1836948115 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9b0001f80 con 0x7fd9b81430c0 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9bcea8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9bcea8640 4 set_mon_vals no callback set 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.698+0000 7fd9b559a640 10 monclient: dump: 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:47.699 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:47.700 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.701 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 10 monclient: shutdown 2024-03-25T18:14:47.701 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:47.701 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1836948115 >> v1:172.21.15.46:6789/0 conn(0x7fd9b81430c0 legacy=0x7fd9b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.701 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1836948115 shutdown_connections 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1836948115 >> 172.21.15.46:0/1836948115 conn(0x7fd9b80b7ea0 msgr2=0x7fd9b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1836948115 shutdown_connections 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.699+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1836948115 wait complete. 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 10 monclient: build_initial_monmap 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 10 monclient: monmap: 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:47.702 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 librados: starting msgr at 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 librados: starting objecter 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 Processor -- start 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 -- start start 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 librados: setting wanted keys 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 1 librados: calling monclient init 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.700+0000 7fd9bcea8640 10 monclient: init 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b81d7df0 con 0x7fd9b80a98f0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b81d7f80 con 0x7fd9b81430c0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b813b5c0 con 0x7fd9b813f710 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:47.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.701+0000 7fd9bcea8640 10 monclient(hunting): authenticate will time out at 7596.963389s 2024-03-25T18:14:47.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.714+0000 7fd9b6d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd9b81430c0 0x7fd9b8135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:42726/0 (socket says 172.21.15.46:42726) 2024-03-25T18:14:47.714 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.714+0000 7fd9b6d9d640 1 -- 172.21.15.46:0/1528277020 learned_addr learned my addr 172.21.15.46:0/1528277020 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:47.715 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.715+0000 7fd9a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd9b81430c0 2024-03-25T18:14:47.715 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.715+0000 7fd9a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.718+0000 7fd9a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd9b813f710 2024-03-25T18:14:47.719 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.718+0000 7fd9a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388097888 0 0) 0x7fd9b81d7f80 con 0x7fd9b81430c0 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:14:47.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.722+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd998004eb0 con 0x7fd9b81430c0 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631020666 0 0) 0x7fd9b813b5c0 con 0x7fd9b813f710 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.726+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b81d7f80 con 0x7fd9b813f710 2024-03-25T18:14:47.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.733+0000 7fd9a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd9b80a98f0 2024-03-25T18:14:47.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.733+0000 7fd9a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.745+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909402229 0 0) 0x7fd9b81d7f80 con 0x7fd9b813f710 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b813b5c0 con 0x7fd9b813f710 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9ac001930 con 0x7fd9b813f710 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.747 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082746676 0 0) 0x7fd9b81d7df0 con 0x7fd9b80a98f0 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:14:47.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.746+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b81d7f80 con 0x7fd9b80a98f0 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3437815371 0 0) 0x7fd998004eb0 con 0x7fd9b81430c0 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b81d7df0 con 0x7fd9b81430c0 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9b0001ec0 con 0x7fd9b81430c0 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.753+0000 7fd9a6ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:47.754 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.755 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:47.755 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:47.755 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:47.755 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:47.755 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:47.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 302122296 0 0) 0x7fd9b81d7df0 con 0x7fd9b81430c0 2024-03-25T18:14:47.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 >> v1:172.21.15.46:6790/0 conn(0x7fd9b813f710 legacy=0x7fd9b8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 >> v1:172.21.15.46:6789/0 conn(0x7fd9b80a98f0 legacy=0x7fd9b81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 1 monclient: found mon.b 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b813b750 con 0x7fd9b81430c0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 10 monclient: _renew_subs 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b8133710 con 0x7fd9b81430c0 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:47.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 10 monclient: _renew_subs 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b81357a0 con 0x7fd9b81430c0 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.764+0000 7fd9bcea8640 1 librados: init done 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.765+0000 7fd9bcea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.765+0000 7fd9bcea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:47.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.765+0000 7fd9bcea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:47.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:47.765+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b8138fd0 con 0x7fd9b81430c0 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9b0004e10 con 0x7fd9b81430c0 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd9b0002100 con 0x7fd9b81430c0 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:48.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9a6ffd640 10 monclient: dump: 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:48.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.049+0000 7fd9b4d99640 4 set_mon_vals no callback set 2024-03-25T18:14:48.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.050+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fd9b001adf0 con 0x7fd9b81430c0 2024-03-25T18:14:48.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.050+0000 7fd9b659c640 10 monclient: get_auth_request con 0x7fd99806df50 auth_method 0 2024-03-25T18:14:48.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.051+0000 7fd9a6ffd640 10 client.4495.objecter ms_handle_connect 0x7fd99806df50 2024-03-25T18:14:48.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fd9b00549a0 con 0x7fd9b81430c0 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 10 client.4495.objecter ms_dispatch 0x7fd9b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9b00502e0 con 0x7fd9b81430c0 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.055+0000 7fd9a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.311+0000 7fd9bcea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.311+0000 7fd9bcea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.311+0000 7fd9bcea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.311+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9b8074cc0 con 0x7fd9b81430c0 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.341+0000 7fd9a6ffd640 1 -- 172.21.15.46:0/1528277020 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fd9b00523e0 con 0x7fd9b81430c0 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.341+0000 7fd9a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.341+0000 7fd9a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:48.343 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 10 librados: watch_flush enter 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 10 librados: watch_flush exit 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 >> v1:172.21.15.164:6800/1612084612 conn(0x7fd99806df50 legacy=0x7fd998070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 10 monclient: shutdown 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.343+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 >> v1:172.21.15.164:6789/0 conn(0x7fd9b81430c0 legacy=0x7fd9b8135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.344+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 shutdown_connections 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.344+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 >> 172.21.15.46:0/1528277020 conn(0x7fd9b80b7ea0 msgr2=0x7fd9b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.344+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 shutdown_connections 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.344+0000 7fd9bcea8640 1 -- 172.21.15.46:0/1528277020 wait complete. 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:48.344+0000 7fd9bcea8640 1 librados: shutdown 2024-03-25T18:14:48.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:55.164 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 10 monclient: monmap: 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:14:55.281962+0000 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:14:55.281962+0000 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.282 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:55.283 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:14:55.283 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:14:55.283 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 1 Processor -- start 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 1 -- start start 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.280+0000 7f9f5cf2b640 10 monclient: init 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f580b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f580b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f5814be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580af940 con 0x7f9f580b1730 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580b0a70 con 0x7f9f580b4d60 2024-03-25T18:14:55.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580b0d50 con 0x7f9f5814be90 2024-03-25T18:14:55.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.281+0000 7f9f5cf2b640 10 monclient(hunting): authenticate will time out at 7604.544354s 2024-03-25T18:14:55.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.286+0000 7f9f5659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9f580b4d60 0x7f9f5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48080/0 (socket says 172.21.15.46:48080) 2024-03-25T18:14:55.288 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.286+0000 7f9f5659c640 1 -- 172.21.15.46:0/3290068406 learned_addr learned my addr 172.21.15.46:0/3290068406 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253230760 0 0) 0x7f9f580b0a70 con 0x7f9f580b4d60 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:14:55.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.306+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f40003520 con 0x7f9f580b4d60 2024-03-25T18:14:55.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445632414 0 0) 0x7f9f580af940 con 0x7f9f580b1730 2024-03-25T18:14:55.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:14:55.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.313+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f580b0a70 con 0x7f9f580b1730 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243491920 0 0) 0x7f9f580b0d50 con 0x7f9f5814be90 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:14:55.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.315+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f580af940 con 0x7f9f5814be90 2024-03-25T18:14:55.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896379693 0 0) 0x7f9f40003520 con 0x7f9f580b4d60 2024-03-25T18:14:55.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 >> v1:172.21.15.46:6790/0 conn(0x7f9f5814be90 legacy=0x7f9f5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.324 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 >> v1:172.21.15.46:6789/0 conn(0x7f9f580b1730 legacy=0x7f9f580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 monclient: found mon.noname-c 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580b1030 con 0x7f9f580b4d60 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9f48001690 con 0x7f9f580b4d60 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5559a640 10 monclient: dump: 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:55.325 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5cf2b640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:14:55.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.322+0000 7f9f5cf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:55.335 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f480017f0 con 0x7f9f580b4d60 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 1 -- 172.21.15.46:0/3290068406 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9f48001c00 con 0x7f9f580b4d60 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5cf2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5cf2b640 4 set_mon_vals no callback set 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.334+0000 7f9f5559a640 10 monclient: dump: 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.336 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:55.337 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:55.337 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:55.337 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 10 monclient: shutdown 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/3290068406 >> v1:172.21.15.164:6789/0 conn(0x7f9f580b4d60 legacy=0x7f9f5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/3290068406 shutdown_connections 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/3290068406 >> 172.21.15.46:0/3290068406 conn(0x7f9f580617a0 msgr2=0x7f9f58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/3290068406 shutdown_connections 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.335+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/3290068406 wait complete. 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 10 monclient: monmap: 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:14:55.338 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.339 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:14:55.339 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:14:55.339 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:14:55.339 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:14:55.339 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 librados: starting msgr at 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 librados: starting objecter 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 Processor -- start 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 -- start start 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 librados: setting wanted keys 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 1 librados: calling monclient init 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.336+0000 7f9f5cf2b640 10 monclient: init 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:55.340 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f580b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f580b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f5814be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58140d00 con 0x7f9f5814be90 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58141cc0 con 0x7f9f580b4d60 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f581d8470 con 0x7f9f580b1730 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.341 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.337+0000 7f9f5cf2b640 10 monclient(hunting): authenticate will time out at 7604.600385s 2024-03-25T18:14:55.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.351+0000 7f9f55d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9f580b4d60 0x7f9f581d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48128/0 (socket says 172.21.15.46:48128) 2024-03-25T18:14:55.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.351+0000 7f9f55d9b640 1 -- 172.21.15.46:0/914178338 learned_addr learned my addr 172.21.15.46:0/914178338 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:14:55.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.357+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f580b4d60 2024-03-25T18:14:55.359 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.357+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.359 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.357+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f5814be90 2024-03-25T18:14:55.359 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.357+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.367 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962991177 0 0) 0x7f9f58140d00 con 0x7f9f5814be90 2024-03-25T18:14:55.368 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.368 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.368 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.368 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:14:55.368 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.366+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f38004eb0 con 0x7f9f5814be90 2024-03-25T18:14:55.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.367+0000 7f9f46ffd640 10 client.?.objecter ms_handle_connect 0x7f9f580b1730 2024-03-25T18:14:55.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.368+0000 7f9f46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69688658 0 0) 0x7f9f58141cc0 con 0x7f9f580b4d60 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:14:55.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.377+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58140d00 con 0x7f9f580b4d60 2024-03-25T18:14:55.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63357627 0 0) 0x7f9f581d8470 con 0x7f9f580b1730 2024-03-25T18:14:55.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:14:55.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.380+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58141cc0 con 0x7f9f580b1730 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164434653 0 0) 0x7f9f38004eb0 con 0x7f9f5814be90 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f581d8470 con 0x7f9f5814be90 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9f50001bd0 con 0x7f9f5814be90 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:55.384 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.382+0000 7f9f46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596923523 0 0) 0x7f9f58140d00 con 0x7f9f580b4d60 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f38004eb0 con 0x7f9f580b4d60 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9f4c0019b0 con 0x7f9f580b4d60 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.384+0000 7f9f46ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.386 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:55.387 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.403+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3465802744 0 0) 0x7f9f581d8470 con 0x7f9f5814be90 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 >> v1:172.21.15.46:6790/0 conn(0x7f9f580b1730 legacy=0x7f9f581405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 >> v1:172.21.15.164:6789/0 conn(0x7f9f580b4d60 legacy=0x7f9f581d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 1 monclient: found mon.a 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f581d8600 con 0x7f9f5814be90 2024-03-25T18:14:55.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 10 monclient: _renew_subs 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5813b3c0 con 0x7f9f5814be90 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 10 monclient: _renew_subs 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f581425a0 con 0x7f9f5814be90 2024-03-25T18:14:55.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.404+0000 7f9f5cf2b640 1 librados: init done 2024-03-25T18:14:55.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.405+0000 7f9f5cf2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:55.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.405+0000 7f9f5cf2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:55.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.405+0000 7f9f5cf2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:14:55.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.405+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f580b2270 con 0x7f9f5814be90 2024-03-25T18:14:55.724 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f50002060 con 0x7f9f5814be90 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9f50003b30 con 0x7f9f5814be90 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f46ffd640 10 monclient: dump: 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:14:55.725 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:14:55.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.723+0000 7f9f54d99640 4 set_mon_vals no callback set 2024-03-25T18:14:55.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.724+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f9f50019b40 con 0x7f9f5814be90 2024-03-25T18:14:55.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.724+0000 7f9f5659c640 10 monclient: get_auth_request con 0x7f9f3806df20 auth_method 0 2024-03-25T18:14:55.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.724+0000 7f9f46ffd640 10 client.4503.objecter ms_handle_connect 0x7f9f3806df20 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f9f50053800 con 0x7f9f5814be90 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 10 client.4503.objecter ms_dispatch 0x7f9f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f5004f140 con 0x7f9f5814be90 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:55.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.731+0000 7f9f46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:55.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.955+0000 7f9f5cf2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:55.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.955+0000 7f9f5cf2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:55.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.955+0000 7f9f5cf2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:14:55.957 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:55.955+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f580627b0 con 0x7f9f5814be90 2024-03-25T18:14:56.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.014+0000 7f9f46ffd640 1 -- 172.21.15.46:0/914178338 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f9f50051240 con 0x7f9f5814be90 2024-03-25T18:14:56.016 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.014+0000 7f9f46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.017 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:14:56.017 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:56.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.014+0000 7f9f46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.016+0000 7f9f5cf2b640 10 librados: watch_flush enter 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.016+0000 7f9f5cf2b640 10 librados: watch_flush exit 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.016+0000 7f9f5cf2b640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.016+0000 7f9f5cf2b640 10 client.4503.objecter successfully canceled tick 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 >> v1:172.21.15.164:6800/1612084612 conn(0x7f9f3806df20 legacy=0x7f9f380703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 10 monclient: shutdown 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:56.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 >> v1:172.21.15.46:6789/0 conn(0x7f9f5814be90 legacy=0x7f9f58134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 shutdown_connections 2024-03-25T18:14:56.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 >> 172.21.15.46:0/914178338 conn(0x7f9f580617a0 msgr2=0x7f9f580b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:56.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 shutdown_connections 2024-03-25T18:14:56.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 -- 172.21.15.46:0/914178338 wait complete. 2024-03-25T18:14:56.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:14:56.017+0000 7f9f5cf2b640 1 librados: shutdown 2024-03-25T18:14:56.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:59.834 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:35.751 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.882 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:35.799 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.885 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:35.802 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.891 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:35.809 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.897 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:06:35.815 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.929 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:06:35.846 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.947 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:35.818 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.952 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:35.823 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:59.977 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:06:35.848 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.839 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:02.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.950+0000 7fe8f8dc4640 10 monclient: get_monmap_and_config 2024-03-25T18:15:02.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.950+0000 7fe8f8dc4640 10 monclient: build_initial_monmap 2024-03-25T18:15:02.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.950+0000 7fe8f8dc4640 10 monclient: monmap: 2024-03-25T18:15:02.952 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:02.952 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:02.952459+0000 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:02.952459+0000 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:02.953 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.951+0000 7fe8f8dc4640 1 Processor -- start 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.951+0000 7fe8f8dc4640 1 -- start start 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.951+0000 7fe8f8dc4640 10 monclient: init 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8f40b4d60 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8f40b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8f414be90 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40af940 con 0x7fe8f40b1730 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40b0a70 con 0x7fe8f414be90 2024-03-25T18:15:02.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40b0d50 con 0x7fe8f40b4d60 2024-03-25T18:15:02.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:02.956 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.952+0000 7fe8f8dc4640 10 monclient(hunting): authenticate will time out at 7612.215361s 2024-03-25T18:15:02.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.960+0000 7fe8ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe8f40b1730 0x7fe8f40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36378/0 (socket says 172.21.15.46:36378) 2024-03-25T18:15:02.962 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.960+0000 7fe8ebfff640 1 -- 172.21.15.46:0/246011186 learned_addr learned my addr 172.21.15.46:0/246011186 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019615563 0 0) 0x7fe8f40b0d50 con 0x7fe8f40b4d60 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:15:03.001 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:02.999+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e0003520 con 0x7fe8f40b4d60 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703271677 0 0) 0x7fe8f40b0a70 con 0x7fe8f414be90 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:15:03.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.056+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f40b0d50 con 0x7fe8f414be90 2024-03-25T18:15:03.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992268331 0 0) 0x7fe8f40af940 con 0x7fe8f40b1730 2024-03-25T18:15:03.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:15:03.063 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.060+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f40b0a70 con 0x7fe8f40b1730 2024-03-25T18:15:03.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834014447 0 0) 0x7fe8f40b0a70 con 0x7fe8f40b1730 2024-03-25T18:15:03.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 >> v1:172.21.15.46:6790/0 conn(0x7fe8f40b4d60 legacy=0x7fe8f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 >> v1:172.21.15.164:6789/0 conn(0x7fe8f414be90 legacy=0x7fe8f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.070 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f40b1030 con 0x7fe8f40b1730 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe8dc0015b0 con 0x7fe8f40b1730 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f1d9b640 10 monclient: dump: 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:03.071 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f8dc4640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:15:03.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.068+0000 7fe8f8dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:03.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8dc001710 con 0x7fe8f40b1730 2024-03-25T18:15:03.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 1 -- 172.21.15.46:0/246011186 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe8dc001b60 con 0x7fe8f40b1730 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f1d9b640 10 monclient: dump: 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:03.096 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:03.097 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f8dc4640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:03.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.093+0000 7fe8f8dc4640 4 set_mon_vals no callback set 2024-03-25T18:15:03.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 10 monclient: shutdown 2024-03-25T18:15:03.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:03.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/246011186 >> v1:172.21.15.46:6789/0 conn(0x7fe8f40b1730 legacy=0x7fe8f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/246011186 shutdown_connections 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/246011186 >> 172.21.15.46:0/246011186 conn(0x7fe8f40617a0 msgr2=0x7fe8f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/246011186 shutdown_connections 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.094+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/246011186 wait complete. 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.095+0000 7fe8f8dc4640 10 monclient: build_initial_monmap 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.095+0000 7fe8f8dc4640 10 monclient: monmap: 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:03.100 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 librados: starting msgr at 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 librados: starting objecter 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 Processor -- start 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 -- start start 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 librados: setting wanted keys 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 1 librados: calling monclient init 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.096+0000 7fe8f8dc4640 10 monclient: init 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8f40b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:03.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8f40b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8f414be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f41324b0 con 0x7fe8f40b1730 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f4132640 con 0x7fe8f40b4d60 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f41327d0 con 0x7fe8f414be90 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:03.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.097+0000 7fe8f8dc4640 10 monclient(hunting): authenticate will time out at 7612.360227s 2024-03-25T18:15:03.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.106+0000 7fe8f259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe8f40b1730 0x7fe8f4144890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36420/0 (socket says 172.21.15.46:36420) 2024-03-25T18:15:03.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.106+0000 7fe8f259c640 1 -- 172.21.15.46:0/1895450119 learned_addr learned my addr 172.21.15.46:0/1895450119 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:03.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.110+0000 7fe8eaffd640 10 client.?.objecter ms_handle_connect 0x7fe8f40b1730 2024-03-25T18:15:03.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.110+0000 7fe8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.114 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.113+0000 7fe8eaffd640 10 client.?.objecter ms_handle_connect 0x7fe8f414be90 2024-03-25T18:15:03.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.113+0000 7fe8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179867640 0 0) 0x7fe8f41327d0 con 0x7fe8f414be90 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:15:03.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.122+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d4004eb0 con 0x7fe8f414be90 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544889697 0 0) 0x7fe8f41324b0 con 0x7fe8f40b1730 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:15:03.126 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.124+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f41327d0 con 0x7fe8f40b1730 2024-03-25T18:15:03.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.131+0000 7fe8eaffd640 10 client.?.objecter ms_handle_connect 0x7fe8f40b4d60 2024-03-25T18:15:03.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.131+0000 7fe8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4143945760 0 0) 0x7fe8d4004eb0 con 0x7fe8f414be90 2024-03-25T18:15:03.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f41324b0 con 0x7fe8f414be90 2024-03-25T18:15:03.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe8ec001bd0 con 0x7fe8f414be90 2024-03-25T18:15:03.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 10 monclient(hunting): dump: 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:03.142 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.139+0000 7fe8eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:03.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638111039 0 0) 0x7fe8f41327d0 con 0x7fe8f40b1730 2024-03-25T18:15:03.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d4004eb0 con 0x7fe8f40b1730 2024-03-25T18:15:03.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe8e4001950 con 0x7fe8f40b1730 2024-03-25T18:15:03.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.141+0000 7fe8eaffd640 10 monclient(hunting): dump: 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:03.144 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467144945 0 0) 0x7fe8f4132640 con 0x7fe8f40b4d60 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:15:03.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.155+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f41327d0 con 0x7fe8f40b4d60 2024-03-25T18:15:03.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2577027280 0 0) 0x7fe8d4004eb0 con 0x7fe8f40b1730 2024-03-25T18:15:03.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 >> v1:172.21.15.46:6790/0 conn(0x7fe8f414be90 legacy=0x7fe8f41d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 >> v1:172.21.15.164:6789/0 conn(0x7fe8f40b4d60 legacy=0x7fe8f41d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 1 monclient: found mon.a 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4138e60 con 0x7fe8f40b1730 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 10 monclient: _renew_subs 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f4139070 con 0x7fe8f40b1730 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:03.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 10 monclient: _renew_subs 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f4137a80 con 0x7fe8f40b1730 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.163+0000 7fe8f8dc4640 1 librados: init done 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.164+0000 7fe8f8dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.164+0000 7fe8f8dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.164+0000 7fe8f8dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.164+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f41d7730 con 0x7fe8f40b1730 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8e4003610 con 0x7fe8f40b1730 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe8e40039a0 con 0x7fe8f40b1730 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8eaffd640 10 monclient: dump: 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:03.473 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:03.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.471+0000 7fe8f159a640 4 set_mon_vals no callback set 2024-03-25T18:15:03.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.472+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fe8e4003b80 con 0x7fe8f40b1730 2024-03-25T18:15:03.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.472+0000 7fe8ebfff640 10 monclient: get_auth_request con 0x7fe8d406df30 auth_method 0 2024-03-25T18:15:03.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.473+0000 7fe8eaffd640 10 client.4509.objecter ms_handle_connect 0x7fe8d406df30 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fe8e40547e0 con 0x7fe8f40b1730 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 10 client.4509.objecter ms_dispatch 0x7fe8f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8e40502f0 con 0x7fe8f40b1730 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.478+0000 7fe8eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:03.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.734+0000 7fe8f8dc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:03.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.734+0000 7fe8f8dc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:03.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.734+0000 7fe8f8dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:03.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.734+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8f413a950 con 0x7fe8f40b1730 2024-03-25T18:15:03.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.766+0000 7fe8eaffd640 1 -- 172.21.15.46:0/1895450119 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fe8f413a950 con 0x7fe8f40b1730 2024-03-25T18:15:03.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.767+0000 7fe8eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:03.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.767+0000 7fe8eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:03.769 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:03.769 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 10 librados: watch_flush enter 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 10 librados: watch_flush exit 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 10 client.4509.objecter successfully canceled tick 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 >> v1:172.21.15.164:6800/1612084612 conn(0x7fe8d406df30 legacy=0x7fe8d40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 10 monclient: shutdown 2024-03-25T18:15:03.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 >> v1:172.21.15.46:6789/0 conn(0x7fe8f40b1730 legacy=0x7fe8f4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 shutdown_connections 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 >> 172.21.15.46:0/1895450119 conn(0x7fe8f40617a0 msgr2=0x7fe8f40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 shutdown_connections 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 -- 172.21.15.46:0/1895450119 wait complete. 2024-03-25T18:15:03.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:03.769+0000 7fe8f8dc4640 1 librados: shutdown 2024-03-25T18:15:03.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:10.591 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:10.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.705+0000 7fc206b8f640 10 monclient: get_monmap_and_config 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.705+0000 7fc206b8f640 10 monclient: build_initial_monmap 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.705+0000 7fc206b8f640 10 monclient: monmap: 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:10.706871+0000 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:10.706871+0000 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:10.707 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.705+0000 7fc206b8f640 1 Processor -- start 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 1 -- start start 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient: init 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc2000a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc20013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc2001430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.706+0000 7fc206b8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc200142520 con 0x7fc20013f710 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.707+0000 7fc206b8f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000b28a0 con 0x7fc2001430c0 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.707+0000 7fc206b8f640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000b2a30 con 0x7fc2000a98f0 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.707+0000 7fc206b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:10.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.707+0000 7fc206b8f640 10 monclient(hunting): authenticate will time out at 7619.969645s 2024-03-25T18:15:10.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.711+0000 7fc20512c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc2001430c0 0x7fc20014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53334/0 (socket says 172.21.15.46:53334) 2024-03-25T18:15:10.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.711+0000 7fc20512c640 1 -- 172.21.15.46:0/1149342200 learned_addr learned my addr 172.21.15.46:0/1149342200 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221810488 0 0) 0x7fc2000b2a30 con 0x7fc2000a98f0 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:15:10.728 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.726+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f0003520 con 0x7fc2000a98f0 2024-03-25T18:15:10.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45842176 0 0) 0x7fc2000b28a0 con 0x7fc2001430c0 2024-03-25T18:15:10.729 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:10.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:10.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:10.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:15:10.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.728+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2000b2a30 con 0x7fc2001430c0 2024-03-25T18:15:10.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.735+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694211033 0 0) 0x7fc2000b2a30 con 0x7fc2001430c0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.735+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 >> v1:172.21.15.46:6790/0 conn(0x7fc2000a98f0 legacy=0x7fc2000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.735+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 >> v1:172.21.15.46:6789/0 conn(0x7fc20013f710 legacy=0x7fc2001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 1 monclient: found mon.noname-c 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2000b44c0 con 0x7fc2001430c0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1f80019f0 con 0x7fc2001430c0 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:10.738 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc1ff7fe640 10 monclient: dump: 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc206b8f640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:15:10.739 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.736+0000 7fc206b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f8001b50 con 0x7fc2001430c0 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 1 -- 172.21.15.46:0/1149342200 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1f8001f60 con 0x7fc2001430c0 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc206b8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc206b8f640 4 set_mon_vals no callback set 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.747+0000 7fc1ff7fe640 10 monclient: dump: 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:10.749 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:10.750 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 10 monclient: shutdown 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 1 -- 172.21.15.46:0/1149342200 >> v1:172.21.15.164:6789/0 conn(0x7fc2001430c0 legacy=0x7fc20014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 1 -- 172.21.15.46:0/1149342200 shutdown_connections 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 1 -- 172.21.15.46:0/1149342200 >> 172.21.15.46:0/1149342200 conn(0x7fc2000b7ea0 msgr2=0x7fc2000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 1 -- 172.21.15.46:0/1149342200 shutdown_connections 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.748+0000 7fc206b8f640 1 -- 172.21.15.46:0/1149342200 wait complete. 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.749+0000 7fc206b8f640 10 monclient: build_initial_monmap 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.749+0000 7fc206b8f640 10 monclient: monmap: 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:10.752 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.749+0000 7fc206b8f640 1 librados: starting msgr at 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.749+0000 7fc206b8f640 1 librados: starting objecter 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 1 Processor -- start 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 1 -- start start 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 1 librados: setting wanted keys 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 1 librados: calling monclient init 2024-03-25T18:15:10.753 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 10 monclient: init 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.750+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc2000a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc20013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc2001430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2001d4980 con 0x7fc2001430c0 2024-03-25T18:15:10.754 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20014dc50 con 0x7fc20013f710 2024-03-25T18:15:10.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2001d4e20 con 0x7fc2000a98f0 2024-03-25T18:15:10.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:10.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.751+0000 7fc206b8f640 10 monclient(hunting): authenticate will time out at 7620.013752s 2024-03-25T18:15:10.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.761+0000 7fc20512c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2001430c0 0x7fc2001d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49684/0 (socket says 172.21.15.46:49684) 2024-03-25T18:15:10.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.761+0000 7fc20512c640 1 -- 172.21.15.46:0/1117720422 learned_addr learned my addr 172.21.15.46:0/1117720422 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:10.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.762+0000 7fc1fd7fa640 10 client.?.objecter ms_handle_connect 0x7fc2001430c0 2024-03-25T18:15:10.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.762+0000 7fc1fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:10.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.769+0000 7fc1fd7fa640 10 client.?.objecter ms_handle_connect 0x7fc2000a98f0 2024-03-25T18:15:10.771 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.769+0000 7fc1fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:10.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.772+0000 7fc1fd7fa640 10 client.?.objecter ms_handle_connect 0x7fc20013f710 2024-03-25T18:15:10.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.772+0000 7fc1fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:10.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178031344 0 0) 0x7fc2001d4980 con 0x7fc2001430c0 2024-03-25T18:15:10.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:10.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:10.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:10.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:15:10.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.775+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e4004eb0 con 0x7fc2001430c0 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595970475 0 0) 0x7fc2001d4e20 con 0x7fc2000a98f0 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:15:10.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.780+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2001d4980 con 0x7fc2000a98f0 2024-03-25T18:15:10.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217709201 0 0) 0x7fc20014dc50 con 0x7fc20013f710 2024-03-25T18:15:10.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:10.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:10.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:10.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:15:10.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.789+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2001d4e20 con 0x7fc20013f710 2024-03-25T18:15:10.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.801+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122298078 0 0) 0x7fc2001d4980 con 0x7fc2000a98f0 2024-03-25T18:15:10.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc20014dc50 con 0x7fc2000a98f0 2024-03-25T18:15:10.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1f40018d0 con 0x7fc2000a98f0 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.802+0000 7fc1fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281532746 0 0) 0x7fc1e4004eb0 con 0x7fc2001430c0 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2001d4980 con 0x7fc2001430c0 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1f8001f20 con 0x7fc2001430c0 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.806+0000 7fc1fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:10.808 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:10.809 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:10.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3054197818 0 0) 0x7fc20014dc50 con 0x7fc2000a98f0 2024-03-25T18:15:10.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 >> v1:172.21.15.164:6789/0 conn(0x7fc20013f710 legacy=0x7fc20014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 >> v1:172.21.15.46:6789/0 conn(0x7fc2001430c0 legacy=0x7fc2001d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 1 monclient: found mon.c 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2001d4fb0 con 0x7fc2000a98f0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc1fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 5 monclient: authenticate success, global_id 4526 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 10 monclient: _renew_subs 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2001d7b00 con 0x7fc2000a98f0 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 10 monclient: _renew_subs 2024-03-25T18:15:10.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20014e0c0 con 0x7fc2000a98f0 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.810+0000 7fc206b8f640 1 librados: init done 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.811+0000 7fc206b8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.811+0000 7fc206b8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.811+0000 7fc206b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:10.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:10.811+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000627b0 con 0x7fc2000a98f0 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f4003610 con 0x7fc2000a98f0 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1f4003a20 con 0x7fc2000a98f0 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 10 monclient: dump: 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:11.130 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1feffd640 4 set_mon_vals no callback set 2024-03-25T18:15:11.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.128+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fc1f4003c40 con 0x7fc2000a98f0 2024-03-25T18:15:11.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.129+0000 7fc1fffff640 10 monclient: get_auth_request con 0x7fc1e406df10 auth_method 0 2024-03-25T18:15:11.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.129+0000 7fc1fd7fa640 10 client.4526.objecter ms_handle_connect 0x7fc1e406df10 2024-03-25T18:15:11.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.135+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fc1f4053530 con 0x7fc2000a98f0 2024-03-25T18:15:11.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.135+0000 7fc1fd7fa640 10 client.4526.objecter ms_dispatch 0x7fc200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.135+0000 7fc1fd7fa640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.135+0000 7fc1fd7fa640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.136+0000 7fc1fd7fa640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.136+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1f404f040 con 0x7fc2000a98f0 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.136+0000 7fc1fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.136+0000 7fc1fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:11.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.388+0000 7fc206b8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.388+0000 7fc206b8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.388+0000 7fc206b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:11.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.388+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc20014e980 con 0x7fc2000a98f0 2024-03-25T18:15:11.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.433+0000 7fc1fd7fa640 1 -- 172.21.15.46:0/1117720422 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fc1f4050f70 con 0x7fc2000a98f0 2024-03-25T18:15:11.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.433+0000 7fc1fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.433+0000 7fc1fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:11.435 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:11.435 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 10 librados: watch_flush enter 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 10 librados: watch_flush exit 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 10 client.4526.objecter successfully canceled tick 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 >> v1:172.21.15.164:6800/1612084612 conn(0x7fc1e406df10 legacy=0x7fc1e40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 10 monclient: shutdown 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 >> v1:172.21.15.46:6790/0 conn(0x7fc2000a98f0 legacy=0x7fc2001d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 shutdown_connections 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 >> 172.21.15.46:0/1117720422 conn(0x7fc2000b7ea0 msgr2=0x7fc20013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 shutdown_connections 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 -- 172.21.15.46:0/1117720422 wait complete. 2024-03-25T18:15:11.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:11.435+0000 7fc206b8f640 1 librados: shutdown 2024-03-25T18:15:11.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:18.256 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.373+0000 7fb83293a640 10 monclient: get_monmap_and_config 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.373+0000 7fb83293a640 10 monclient: build_initial_monmap 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.373+0000 7fb83293a640 10 monclient: monmap: 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:18.374987+0000 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:18.374987+0000 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:18.375 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.376 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:18.376 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:18.376 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:18.376 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 1 Processor -- start 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 1 -- start start 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient: init 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-a con 0x7fb82c0b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-c con 0x7fb82c0b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-b con 0x7fb82c14be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0af940 con 0x7fb82c0b4d60 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0b0a70 con 0x7fb82c0b1730 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0b0d50 con 0x7fb82c14be90 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:18.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.374+0000 7fb83293a640 10 monclient(hunting): authenticate will time out at 7627.637437s 2024-03-25T18:15:18.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.382+0000 7fb82b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b1730 0x7fb82c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:32994/0 (socket says 172.21.15.46:32994) 2024-03-25T18:15:18.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.382+0000 7fb82b7fe640 1 -- 172.21.15.46:0/4053446132 learned_addr learned my addr 172.21.15.46:0/4053446132 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:18.392 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.390+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366627652 0 0) 0x7fb82c0b0a70 con 0x7fb82c0b1730 2024-03-25T18:15:18.392 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.391+0000 7fb82affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.392 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.391+0000 7fb82affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.392 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.391+0000 7fb82affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.391+0000 7fb82affd640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:15:18.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.391+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb814003520 con 0x7fb82c0b1730 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115273071 0 0) 0x7fb82c0b0d50 con 0x7fb82c14be90 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:15:18.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c0b0a70 con 0x7fb82c14be90 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190752882 0 0) 0x7fb814003520 con 0x7fb82c0b1730 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 >> v1:172.21.15.46:6790/0 conn(0x7fb82c14be90 legacy=0x7fb82c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 >> v1:172.21.15.46:6789/0 conn(0x7fb82c0b4d60 legacy=0x7fb82c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 monclient: found mon.noname-c 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0b1030 con 0x7fb82c0b1730 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb820001610 con 0x7fb82c0b1730 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb83293a640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb83293a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb82affd640 10 monclient: dump: 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.408 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.404+0000 7fb83293a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb820001770 con 0x7fb82c0b1730 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 1 -- 172.21.15.46:0/4053446132 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb820001bc0 con 0x7fb82c0b1730 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:18.415 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb82affd640 10 monclient: dump: 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb83293a640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:18.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.413+0000 7fb83293a640 4 set_mon_vals no callback set 2024-03-25T18:15:18.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 10 monclient: shutdown 2024-03-25T18:15:18.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:18.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 1 -- 172.21.15.46:0/4053446132 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b1730 legacy=0x7fb82c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 1 -- 172.21.15.46:0/4053446132 shutdown_connections 2024-03-25T18:15:18.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 1 -- 172.21.15.46:0/4053446132 >> 172.21.15.46:0/4053446132 conn(0x7fb82c0617a0 msgr2=0x7fb82c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 1 -- 172.21.15.46:0/4053446132 shutdown_connections 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 1 -- 172.21.15.46:0/4053446132 wait complete. 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.415+0000 7fb83293a640 10 monclient: build_initial_monmap 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 10 monclient: monmap: 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:18.418 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 librados: starting msgr at 2024-03-25T18:15:18.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 librados: starting objecter 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 Processor -- start 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 -- start start 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 librados: setting wanted keys 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 1 librados: calling monclient init 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.416+0000 7fb83293a640 10 monclient: init 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-a con 0x7fb82c0b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-b con 0x7fb82c0b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:18.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient(hunting): picked mon.noname-c con 0x7fb82c14be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:18.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c141fc0 con 0x7fb82c0b1730 2024-03-25T18:15:18.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c1dac50 con 0x7fb82c0b4d60 2024-03-25T18:15:18.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c140d00 con 0x7fb82c14be90 2024-03-25T18:15:18.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:18.421 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.417+0000 7fb83293a640 10 monclient(hunting): authenticate will time out at 7627.680310s 2024-03-25T18:15:18.425 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.424+0000 7fb82b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 0x7fb82c1db170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:33030/0 (socket says 172.21.15.46:33030) 2024-03-25T18:15:18.426 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.424+0000 7fb82b7fe640 1 -- 172.21.15.46:0/2273885709 learned_addr learned my addr 172.21.15.46:0/2273885709 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:18.426 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.425+0000 7fb828ff9640 10 client.?.objecter ms_handle_connect 0x7fb82c0b4d60 2024-03-25T18:15:18.426 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.425+0000 7fb828ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.427+0000 7fb828ff9640 10 client.?.objecter ms_handle_connect 0x7fb82c0b1730 2024-03-25T18:15:18.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.427+0000 7fb828ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.428+0000 7fb82b7fe640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 legacy=0x7fb82c1db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:15:18.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.428+0000 7fb82b7fe640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 legacy=0x7fb82c1db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:15:18.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.428+0000 7fb82b7fe640 1 --1- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 0x7fb82c1db170 :-1 s=OPENED pgs=46193 cs=1 l=1).handle_message read tag failed 2024-03-25T18:15:18.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.428+0000 7fb82b7fe640 1 --1- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 0x7fb82c1db170 :-1 s=OPENED pgs=46193 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:15:18.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.428+0000 7fb828ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.164:6789/0 2024-03-25T18:15:18.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.436+0000 7fb828ff9640 10 client.?.objecter ms_handle_connect 0x7fb82c14be90 2024-03-25T18:15:18.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.436+0000 7fb828ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525150483 0 0) 0x7fb82c140d00 con 0x7fb82c14be90 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:15:18.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.444+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb818004eb0 con 0x7fb82c14be90 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648280605 0 0) 0x7fb82c141fc0 con 0x7fb82c0b1730 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:15:18.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.446+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c140d00 con 0x7fb82c0b1730 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406509473 0 0) 0x7fb818004eb0 con 0x7fb82c14be90 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c141fc0 con 0x7fb82c14be90 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb824001b50 con 0x7fb82c14be90 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:18.477 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.475+0000 7fb828ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:18.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.478+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423781328 0 0) 0x7fb82c140d00 con 0x7fb82c0b1730 2024-03-25T18:15:18.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.478+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb818004eb0 con 0x7fb82c0b1730 2024-03-25T18:15:18.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.479+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb81c001870 con 0x7fb82c0b1730 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.479+0000 7fb828ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.479+0000 7fb828ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.479+0000 7fb828ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:18.481 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4273261991 0 0) 0x7fb818004eb0 con 0x7fb82c0b1730 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.46:6790/0 conn(0x7fb82c14be90 legacy=0x7fb82c134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6789/0 conn(0x7fb82c0b4d60 legacy=0x7fb82c1db170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 1 monclient: found mon.a 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c1423e0 con 0x7fb82c0b1730 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:18.496 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 10 monclient: _renew_subs 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c13b470 con 0x7fb82c0b1730 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 10 monclient: _renew_subs 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c1dba60 con 0x7fb82c0b1730 2024-03-25T18:15:18.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.494+0000 7fb83293a640 1 librados: init done 2024-03-25T18:15:18.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.495+0000 7fb83293a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:18.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.495+0000 7fb83293a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:18.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.495+0000 7fb83293a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:18.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.495+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c1401f0 con 0x7fb82c0b1730 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb81c001d60 con 0x7fb82c0b1730 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb81c0037b0 con 0x7fb82c0b1730 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb828ff9640 10 monclient: dump: 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:18.816 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:18.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.814+0000 7fb82a7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:18.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.815+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb81c0039d0 con 0x7fb82c0b1730 2024-03-25T18:15:18.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.816+0000 7fb82b7fe640 10 monclient: get_auth_request con 0x7fb81806dd50 auth_method 0 2024-03-25T18:15:18.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.816+0000 7fb828ff9640 10 client.4518.objecter ms_handle_connect 0x7fb81806dd50 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb81c0535b0 con 0x7fb82c0b1730 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 10 client.4518.objecter ms_dispatch 0x7fb82c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T18:15:18.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb81c04eef0 con 0x7fb82c0b1730 2024-03-25T18:15:18.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:18.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:18.822+0000 7fb828ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.067+0000 7fb83293a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.067+0000 7fb83293a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.067+0000 7fb83293a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.067+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb82c1dade0 con 0x7fb82c0b1730 2024-03-25T18:15:19.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.163+0000 7fb828ff9640 1 -- 172.21.15.46:0/2273885709 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb81c050ff0 con 0x7fb82c0b1730 2024-03-25T18:15:19.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.163+0000 7fb828ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.163+0000 7fb828ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:19.166 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:19.166 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:19.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 10 librados: watch_flush enter 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 10 librados: watch_flush exit 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 10 client.4518.objecter successfully canceled tick 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb81806dd50 legacy=0x7fb818070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 10 monclient: shutdown 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 >> v1:172.21.15.46:6789/0 conn(0x7fb82c0b1730 legacy=0x7fb82c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 shutdown_connections 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 >> 172.21.15.46:0/2273885709 conn(0x7fb82c0617a0 msgr2=0x7fb82c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 shutdown_connections 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.166+0000 7fb83293a640 1 -- 172.21.15.46:0/2273885709 wait complete. 2024-03-25T18:15:19.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:19.167+0000 7fb83293a640 1 librados: shutdown 2024-03-25T18:15:19.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:24.977 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:00.894 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:24.979 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:00.897 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.013 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:00.931 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.045 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:00.962 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.045 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:00.915 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.052 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:00.969 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.057 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:00.928 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.099 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:01.017 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.103 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:00.974 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:25.989 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.090+0000 7f628b4a8640 10 monclient: get_monmap_and_config 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.090+0000 7f628b4a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.090+0000 7f628b4a8640 10 monclient: monmap: 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:26.091583+0000 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:26.091583+0000 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:26.092 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.091+0000 7f628b4a8640 1 Processor -- start 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.091+0000 7f628b4a8640 1 -- start start 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.091+0000 7f628b4a8640 10 monclient: init 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f62840a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f628413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f62841430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6284142520 con 0x7f62840a98f0 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840b28a0 con 0x7f628413f710 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840b2a30 con 0x7f62841430c0 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.092+0000 7f628b4a8640 10 monclient(hunting): authenticate will time out at 7635.354159s 2024-03-25T18:15:26.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.093+0000 7f6289244640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f62840a98f0 0x7f62840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56968/0 (socket says 172.21.15.46:56968) 2024-03-25T18:15:26.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.093+0000 7f6289244640 1 -- 172.21.15.46:0/2846826233 learned_addr learned my addr 172.21.15.46:0/2846826233 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960962012 0 0) 0x7f62840b2a30 con 0x7f62841430c0 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:15:26.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.123+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270003520 con 0x7f62841430c0 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078850934 0 0) 0x7f6270003520 con 0x7f62841430c0 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 >> v1:172.21.15.164:6789/0 conn(0x7f628413f710 legacy=0x7f62841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 >> v1:172.21.15.46:6789/0 conn(0x7f62840a98f0 legacy=0x7f62840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 monclient: found mon.noname-b 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.147 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840b44c0 con 0x7f62841430c0 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f627c001a10 con 0x7f62841430c0 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f628b4a8640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f628b4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f627bfff640 10 monclient: dump: 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.148 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:26.149 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:26.149 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:26.149 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:26.149 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.146+0000 7f628b4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:26.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f627c001b70 con 0x7f62841430c0 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 1 -- 172.21.15.46:0/2846826233 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f627c001f80 con 0x7f62841430c0 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f627bfff640 10 monclient: dump: 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:26.211 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:26.212 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:26.212 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f628b4a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:26.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.210+0000 7f628b4a8640 4 set_mon_vals no callback set 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 10 monclient: shutdown 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 1 -- 172.21.15.46:0/2846826233 >> v1:172.21.15.46:6790/0 conn(0x7f62841430c0 legacy=0x7f628414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 1 -- 172.21.15.46:0/2846826233 shutdown_connections 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 1 -- 172.21.15.46:0/2846826233 >> 172.21.15.46:0/2846826233 conn(0x7f62840b7ea0 msgr2=0x7f62840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 1 -- 172.21.15.46:0/2846826233 shutdown_connections 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 1 -- 172.21.15.46:0/2846826233 wait complete. 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.211+0000 7f628b4a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.214 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient: monmap: 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 librados: starting msgr at 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 librados: starting objecter 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 Processor -- start 2024-03-25T18:15:26.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 -- start start 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 librados: setting wanted keys 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 librados: calling monclient init 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient: init 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f62840a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f628413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f62841430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62841369d0 con 0x7f62840a98f0 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6284133a60 con 0x7f62841430c0 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.212+0000 7f628b4a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62840ab9e0 con 0x7f628413f710 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.213+0000 7f628b4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.213+0000 7f628b4a8640 10 monclient(hunting): authenticate will time out at 7635.474551s 2024-03-25T18:15:26.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.214+0000 7f6289a45640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f62841430c0 0x7f6284133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:56216/0 (socket says 172.21.15.46:56216) 2024-03-25T18:15:26.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.214+0000 7f6289a45640 1 -- 172.21.15.46:0/1519514753 learned_addr learned my addr 172.21.15.46:0/1519514753 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:26.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.215+0000 7f6279ffb640 10 client.?.objecter ms_handle_connect 0x7f62841430c0 2024-03-25T18:15:26.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.215+0000 7f6279ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.221+0000 7f6279ffb640 10 client.?.objecter ms_handle_connect 0x7f628413f710 2024-03-25T18:15:26.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.221+0000 7f6279ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426884441 0 0) 0x7f62840ab9e0 con 0x7f628413f710 2024-03-25T18:15:26.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:15:26.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.233+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268004eb0 con 0x7f628413f710 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 624776232 0 0) 0x7f6268004eb0 con 0x7f628413f710 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62840ab9e0 con 0x7f628413f710 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6274001930 con 0x7f628413f710 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:26.257 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.256+0000 7f6279ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:26.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.265+0000 7f6279ffb640 10 client.?.objecter ms_handle_connect 0x7f62840a98f0 2024-03-25T18:15:26.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.265+0000 7f6279ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1502993139 0 0) 0x7f62840ab9e0 con 0x7f628413f710 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 >> v1:172.21.15.164:6789/0 conn(0x7f62841430c0 legacy=0x7f6284133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 >> v1:172.21.15.46:6789/0 conn(0x7f62840a98f0 legacy=0x7f6284132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 1 monclient: found mon.c 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840abcc0 con 0x7f628413f710 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f6279ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f628b4a8640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f628b4a8640 10 monclient: _renew_subs 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f628b4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.266+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62840abfa0 con 0x7f628413f710 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 monclient: _renew_subs 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6284137e40 con 0x7f628413f710 2024-03-25T18:15:26.269 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 1 librados: init done 2024-03-25T18:15:26.270 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.270 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.270 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.270 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.267+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6284139060 con 0x7f628413f710 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6274003610 con 0x7f628413f710 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f62740039e0 con 0x7f628413f710 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f6279ffb640 10 monclient: dump: 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.590 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.589+0000 7f627b7fe640 4 set_mon_vals no callback set 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.590+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f6274003c00 con 0x7f628413f710 2024-03-25T18:15:26.591 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.590+0000 7f6289244640 10 monclient: get_auth_request con 0x7f626806dd00 auth_method 0 2024-03-25T18:15:26.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.591+0000 7f6279ffb640 10 client.4538.objecter ms_handle_connect 0x7f626806dd00 2024-03-25T18:15:26.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.597+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f6274053670 con 0x7f628413f710 2024-03-25T18:15:26.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.597+0000 7f6279ffb640 10 client.4538.objecter ms_dispatch 0x7f6284000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.597+0000 7f6279ffb640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.597+0000 7f6279ffb640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.598+0000 7f6279ffb640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.598+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62740b08b0 con 0x7f628413f710 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.598+0000 7f6279ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.599 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.598+0000 7f6279ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:26.862 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.862+0000 7f628b4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:26.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.862+0000 7f628b4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:26.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.862+0000 7f628b4a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:26.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.862+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62840627b0 con 0x7f628413f710 2024-03-25T18:15:26.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.901+0000 7f6279ffb640 1 -- 172.21.15.46:0/1519514753 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f627404efb0 con 0x7f628413f710 2024-03-25T18:15:26.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.901+0000 7f6279ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:26.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.901+0000 7f6279ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:26.902 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:26.902 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 10 librados: watch_flush enter 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 10 librados: watch_flush exit 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 >> v1:172.21.15.164:6800/1612084612 conn(0x7f626806dd00 legacy=0x7f62680701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 10 monclient: shutdown 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 >> v1:172.21.15.46:6790/0 conn(0x7f628413f710 legacy=0x7f62841362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 shutdown_connections 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 >> 172.21.15.46:0/1519514753 conn(0x7f62840b7ea0 msgr2=0x7f6284074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 shutdown_connections 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 -- 172.21.15.46:0/1519514753 wait complete. 2024-03-25T18:15:26.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:26.903+0000 7f628b4a8640 1 librados: shutdown 2024-03-25T18:15:26.917 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:33.722 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:33.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.837+0000 7fb2dc68d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.837+0000 7fb2dc68d640 10 monclient: build_initial_monmap 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.837+0000 7fb2dc68d640 10 monclient: monmap: 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:33.838630+0000 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:33.838630+0000 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:33.839 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:33.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.838+0000 7fb2dc68d640 1 Processor -- start 2024-03-25T18:15:33.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.838+0000 7fb2dc68d640 1 -- start start 2024-03-25T18:15:33.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.838+0000 7fb2dc68d640 10 monclient: init 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d40a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4142520 con 0x7fb2d40a98f0 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40b28a0 con 0x7fb2d413f710 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40b2a30 con 0x7fb2d41430c0 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:33.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.839+0000 7fb2dc68d640 10 monclient(hunting): authenticate will time out at 7643.101137s 2024-03-25T18:15:33.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.849+0000 7fb2da429640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2d40a98f0 0x7fb2d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49794/0 (socket says 172.21.15.46:49794) 2024-03-25T18:15:33.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.849+0000 7fb2da429640 1 -- 172.21.15.46:0/1373996301 learned_addr learned my addr 172.21.15.46:0/1373996301 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:33.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156278642 0 0) 0x7fb2d40b28a0 con 0x7fb2d413f710 2024-03-25T18:15:33.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 10 monclient(hunting): _negotiate 2024-03-25T18:15:33.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:33.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:33.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:15:33.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.858+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2bc003520 con 0x7fb2d413f710 2024-03-25T18:15:33.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256607360 0 0) 0x7fb2d4142520 con 0x7fb2d40a98f0 2024-03-25T18:15:33.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 10 monclient(hunting): _negotiate 2024-03-25T18:15:33.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:33.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:33.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:15:33.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.865+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d40b28a0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198582287 0 0) 0x7fb2d40b2a30 con 0x7fb2d41430c0 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 10 monclient(hunting): _negotiate 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:15:33.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.874+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4142520 con 0x7fb2d41430c0 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2304868299 0 0) 0x7fb2d40b28a0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 >> v1:172.21.15.46:6790/0 conn(0x7fb2d41430c0 legacy=0x7fb2d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 >> v1:172.21.15.164:6789/0 conn(0x7fb2d413f710 legacy=0x7fb2d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: _finish_hunting 0 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 monclient: found mon.noname-a 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:33.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40b44c0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: _finish_auth 0 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: _check_auth_tickets 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb2c40019f0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2d9427640 10 monclient: dump: 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:33.878 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2dc68d640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:15:33.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.876+0000 7fb2dc68d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c4001bd0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2dc68d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 1 -- 172.21.15.46:0/1373996301 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb2c4001fe0 con 0x7fb2d40a98f0 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:33.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2dc68d640 4 set_mon_vals no callback set 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.893+0000 7fb2d9427640 10 monclient: dump: 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:33.895 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 10 monclient: shutdown 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 1 -- 172.21.15.46:0/1373996301 >> v1:172.21.15.46:6789/0 conn(0x7fb2d40a98f0 legacy=0x7fb2d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 1 -- 172.21.15.46:0/1373996301 shutdown_connections 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 1 -- 172.21.15.46:0/1373996301 >> 172.21.15.46:0/1373996301 conn(0x7fb2d40b7ea0 msgr2=0x7fb2d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:33.897 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 1 -- 172.21.15.46:0/1373996301 shutdown_connections 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.894+0000 7fb2dc68d640 1 -- 172.21.15.46:0/1373996301 wait complete. 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 10 monclient: build_initial_monmap 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 10 monclient: monmap: 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:33.898 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 librados: starting msgr at 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 librados: starting objecter 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 Processor -- start 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 -- start start 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 librados: setting wanted keys 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 1 librados: calling monclient init 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.895+0000 7fb2dc68d640 10 monclient: init 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d40a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4137030 con 0x7fb2d40a98f0 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d41398a0 con 0x7fb2d413f710 2024-03-25T18:15:33.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d41d4c90 con 0x7fb2d41430c0 2024-03-25T18:15:33.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:33.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.896+0000 7fb2dc68d640 10 monclient(hunting): authenticate will time out at 7643.158240s 2024-03-25T18:15:33.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.907+0000 7fb2d9c28640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb2d413f710 0x7fb2d40aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:35106/0 (socket says 172.21.15.46:35106) 2024-03-25T18:15:33.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.907+0000 7fb2d9c28640 1 -- 172.21.15.46:0/2578436836 learned_addr learned my addr 172.21.15.46:0/2578436836 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:33.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.909+0000 7fb2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2d413f710 2024-03-25T18:15:33.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.909+0000 7fb2d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:33.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.912+0000 7fb2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2d41430c0 2024-03-25T18:15:33.913 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.912+0000 7fb2d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:33.915 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898492196 0 0) 0x7fb2d41398a0 con 0x7fb2d413f710 2024-03-25T18:15:33.915 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:33.916 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:33.916 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:33.916 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:15:33.916 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.915+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0004eb0 con 0x7fb2d413f710 2024-03-25T18:15:33.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.921+0000 7fb2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2d40a98f0 2024-03-25T18:15:33.922 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.921+0000 7fb2d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:33.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303704156 0 0) 0x7fb2d41d4c90 con 0x7fb2d41430c0 2024-03-25T18:15:33.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:33.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:33.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:33.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:15:33.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.927+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d41398a0 con 0x7fb2d41430c0 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2023926722 0 0) 0x7fb2c0004eb0 con 0x7fb2d413f710 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d41d4c90 con 0x7fb2d413f710 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb2c8001990 con 0x7fb2d413f710 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:33.930 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:33.931 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.929+0000 7fb2d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:33.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744857540 0 0) 0x7fb2d41d4c90 con 0x7fb2d413f710 2024-03-25T18:15:33.937 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 >> v1:172.21.15.46:6790/0 conn(0x7fb2d41430c0 legacy=0x7fb2d41d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 >> v1:172.21.15.46:6789/0 conn(0x7fb2d40a98f0 legacy=0x7fb2d41d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 1 monclient: found mon.b 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40b16b0 con 0x7fb2d413f710 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 10 monclient: _renew_subs 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:33.938 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d40b0c10 con 0x7fb2d413f710 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 10 monclient: _renew_subs 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d40b07f0 con 0x7fb2d413f710 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.936+0000 7fb2dc68d640 1 librados: init done 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.937+0000 7fb2dc68d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:33.939 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.937+0000 7fb2dc68d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:33.940 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.937+0000 7fb2dc68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:33.940 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:33.937+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d41d4e20 con 0x7fb2d413f710 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c80035f0 con 0x7fb2d413f710 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb2c80039c0 con 0x7fb2d413f710 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d2ffd640 10 monclient: dump: 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:34.210 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.209+0000 7fb2d8c26640 4 set_mon_vals no callback set 2024-03-25T18:15:34.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.210+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb2c8003be0 con 0x7fb2d413f710 2024-03-25T18:15:34.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.210+0000 7fb2da429640 10 monclient: get_auth_request con 0x7fb2c006dd10 auth_method 0 2024-03-25T18:15:34.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.210+0000 7fb2d2ffd640 10 client.4525.objecter ms_handle_connect 0x7fb2c006dd10 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.218+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb2c8054790 con 0x7fb2d413f710 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.218+0000 7fb2d2ffd640 10 client.4525.objecter ms_dispatch 0x7fb2d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.218+0000 7fb2d2ffd640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.218+0000 7fb2d2ffd640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.218+0000 7fb2d2ffd640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:15:34.219 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.219+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2c80502a0 con 0x7fb2d413f710 2024-03-25T18:15:34.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.219+0000 7fb2d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.219+0000 7fb2d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:34.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.478+0000 7fb2dc68d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.478+0000 7fb2dc68d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.478+0000 7fb2dc68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:34.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.478+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2d40b0f10 con 0x7fb2d413f710 2024-03-25T18:15:34.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.498+0000 7fb2d2ffd640 1 -- 172.21.15.46:0/2578436836 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb2c80521d0 con 0x7fb2d413f710 2024-03-25T18:15:34.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.498+0000 7fb2d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.498+0000 7fb2d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:34.499 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:34.500 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 10 librados: watch_flush enter 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 10 librados: watch_flush exit 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb2c006dd10 legacy=0x7fb2c00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 10 monclient: shutdown 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 >> v1:172.21.15.164:6789/0 conn(0x7fb2d413f710 legacy=0x7fb2d40aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 shutdown_connections 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 >> 172.21.15.46:0/2578436836 conn(0x7fb2d40b7ea0 msgr2=0x7fb2d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 shutdown_connections 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 -- 172.21.15.46:0/2578436836 wait complete. 2024-03-25T18:15:34.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:34.500+0000 7fb2dc68d640 1 librados: shutdown 2024-03-25T18:15:34.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:41.319 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:41.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.436+0000 7f65354b3640 10 monclient: get_monmap_and_config 2024-03-25T18:15:41.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.437+0000 7f65354b3640 10 monclient: build_initial_monmap 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.437+0000 7f65354b3640 10 monclient: monmap: 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:41.437706+0000 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:41.437706+0000 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.437+0000 7f65354b3640 1 Processor -- start 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.437+0000 7f65354b3640 1 -- start start 2024-03-25T18:15:41.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.437+0000 7f65354b3640 10 monclient: init 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-c con 0x7f65300b1ef0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-b con 0x7f65301473a0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6530150070 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300b2570 con 0x7f6530150070 2024-03-25T18:15:41.440 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530146c20 con 0x7f65300b1ef0 2024-03-25T18:15:41.441 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.438+0000 7f65354b3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530146e80 con 0x7f65301473a0 2024-03-25T18:15:41.441 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.439+0000 7f65354b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:41.441 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.439+0000 7f65354b3640 10 monclient(hunting): authenticate will time out at 7650.700548s 2024-03-25T18:15:41.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.446+0000 7f652effd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f65300b1ef0 0x7f65300b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:39674/0 (socket says 172.21.15.46:39674) 2024-03-25T18:15:41.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.446+0000 7f652effd640 1 -- 172.21.15.46:0/2681067684 learned_addr learned my addr 172.21.15.46:0/2681067684 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367522941 0 0) 0x7f6530146c20 con 0x7f65300b1ef0 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:15:41.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.450+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6518003520 con 0x7f65300b1ef0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.465+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817826584 0 0) 0x7f6518003520 con 0x7f65300b1ef0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 >> v1:172.21.15.46:6790/0 conn(0x7f65301473a0 legacy=0x7f6530143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 >> v1:172.21.15.46:6789/0 conn(0x7f6530150070 legacy=0x7f6530152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 1 monclient: found mon.noname-c 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653006e1b0 con 0x7f65300b1ef0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:41.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6520001690 con 0x7f65300b1ef0 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f65354b3640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f65354b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f652dffb640 10 monclient: dump: 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.466+0000 7f65354b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:41.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6520001860 con 0x7f65300b1ef0 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 1 -- 172.21.15.46:0/2681067684 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6520001cb0 con 0x7f65300b1ef0 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f65354b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f65354b3640 4 set_mon_vals no callback set 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f652dffb640 10 monclient: dump: 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:41.483 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:41.484 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:41.484 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f65354b3640 10 monclient: shutdown 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f65354b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.482+0000 7f65354b3640 1 -- 172.21.15.46:0/2681067684 >> v1:172.21.15.164:6789/0 conn(0x7f65300b1ef0 legacy=0x7f65300b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 1 -- 172.21.15.46:0/2681067684 shutdown_connections 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 1 -- 172.21.15.46:0/2681067684 >> 172.21.15.46:0/2681067684 conn(0x7f65300b7ea0 msgr2=0x7f65300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 1 -- 172.21.15.46:0/2681067684 shutdown_connections 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 1 -- 172.21.15.46:0/2681067684 wait complete. 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 10 monclient: build_initial_monmap 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.483+0000 7f65354b3640 10 monclient: monmap: 2024-03-25T18:15:41.485 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 librados: starting msgr at 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 librados: starting objecter 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 Processor -- start 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 -- start start 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 librados: setting wanted keys 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 1 librados: calling monclient init 2024-03-25T18:15:41.486 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.484+0000 7f65354b3640 10 monclient: init 2024-03-25T18:15:41.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:41.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:41.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-b con 0x7f65300b1ef0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:41.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-c con 0x7f65301473a0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:41.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6530150070 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:41.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65301db620 con 0x7f6530150070 2024-03-25T18:15:41.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65301db870 con 0x7f65300b1ef0 2024-03-25T18:15:41.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65301dc4a0 con 0x7f65301473a0 2024-03-25T18:15:41.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:41.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.485+0000 7f65354b3640 10 monclient(hunting): authenticate will time out at 7650.746970s 2024-03-25T18:15:41.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.498+0000 7f652e7fc640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f65301473a0 0x7f6530137040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:49670/0 (socket says 172.21.15.46:49670) 2024-03-25T18:15:41.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.498+0000 7f652e7fc640 1 -- 172.21.15.46:0/1994358540 learned_addr learned my addr 172.21.15.46:0/1994358540 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:41.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.499+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f65301473a0 2024-03-25T18:15:41.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.499+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.502+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f6530150070 2024-03-25T18:15:41.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.502+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146855910 0 0) 0x7f65301dc4a0 con 0x7f65301473a0 2024-03-25T18:15:41.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:15:41.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.508+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c004eb0 con 0x7f65301473a0 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602343042 0 0) 0x7f65301db620 con 0x7f6530150070 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:15:41.513 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.512+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65301dc4a0 con 0x7f6530150070 2024-03-25T18:15:41.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.515+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f65300b1ef0 2024-03-25T18:15:41.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.515+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.519+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934941029 0 0) 0x7f651c004eb0 con 0x7f65301473a0 2024-03-25T18:15:41.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.519+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65301db620 con 0x7f65301473a0 2024-03-25T18:15:41.526 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905885501 0 0) 0x7f65301db870 con 0x7f65300b1ef0 2024-03-25T18:15:41.526 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.526 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.526 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.526 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:15:41.527 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.525+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c004eb0 con 0x7f65300b1ef0 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f65240019f0 con 0x7f65301473a0 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient(hunting): dump: 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:41.537 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1581730565 0 0) 0x7f65301db620 con 0x7f65301473a0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 >> v1:172.21.15.164:6789/0 conn(0x7f65300b1ef0 legacy=0x7f65301dbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 >> v1:172.21.15.46:6789/0 conn(0x7f6530150070 legacy=0x7f6530145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 monclient: found mon.c 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6530137750 con 0x7f65301473a0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 10 monclient: _renew_subs 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:41.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65301379a0 con 0x7f65301473a0 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 10 monclient: _renew_subs 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65301417a0 con 0x7f65301473a0 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.535+0000 7f65354b3640 1 librados: init done 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.536+0000 7f65354b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.537+0000 7f65354b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.537+0000 7f65354b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:41.539 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.537+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65300627b0 con 0x7f65301473a0 2024-03-25T18:15:41.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.540+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65240035f0 con 0x7f65301473a0 2024-03-25T18:15:41.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.540+0000 7f65177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:41.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.541+0000 7f652d7fa640 4 set_mon_vals no callback set 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.841+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f6524003a40 con 0x7f65301473a0 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.841+0000 7f65177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.841+0000 7f65177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.841+0000 7f65177fe640 10 monclient: dump: 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:41.842 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:41.843 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:41.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.841+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f6524003c20 con 0x7f65301473a0 2024-03-25T18:15:41.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.842+0000 7f652effd640 10 monclient: get_auth_request con 0x7f651c06de20 auth_method 0 2024-03-25T18:15:41.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.843+0000 7f65177fe640 10 client.4547.objecter ms_handle_connect 0x7f651c06de20 2024-03-25T18:15:41.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.847+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f6524054840 con 0x7f65301473a0 2024-03-25T18:15:41.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.847+0000 7f65177fe640 10 client.4547.objecter ms_dispatch 0x7f6530000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.847+0000 7f65177fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.847+0000 7f65177fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.847+0000 7f65177fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.848+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6524050350 con 0x7f65301473a0 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.848+0000 7f65177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:41.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:41.848+0000 7f65177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:42.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.077+0000 7f65354b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.077+0000 7f65354b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.077+0000 7f65354b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:15:42.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.077+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6530069f40 con 0x7f65301473a0 2024-03-25T18:15:42.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.106+0000 7f65177fe640 1 -- 172.21.15.46:0/1994358540 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f6524052280 con 0x7f65301473a0 2024-03-25T18:15:42.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.106+0000 7f65177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.106+0000 7f65177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:42.107 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:42.108 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.109+0000 7f65354b3640 10 librados: watch_flush enter 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.109+0000 7f65354b3640 10 librados: watch_flush exit 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.109+0000 7f65354b3640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.109+0000 7f65354b3640 10 client.4547.objecter successfully canceled tick 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 >> v1:172.21.15.164:6800/1612084612 conn(0x7f651c06de20 legacy=0x7f651c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 10 monclient: shutdown 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 >> v1:172.21.15.46:6790/0 conn(0x7f65301473a0 legacy=0x7f6530137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 shutdown_connections 2024-03-25T18:15:42.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 >> 172.21.15.46:0/1994358540 conn(0x7f65300b7ea0 msgr2=0x7f653013a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:42.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 shutdown_connections 2024-03-25T18:15:42.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 -- 172.21.15.46:0/1994358540 wait complete. 2024-03-25T18:15:42.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:42.110+0000 7f65354b3640 1 librados: shutdown 2024-03-25T18:15:42.123 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:44.991 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:20.908 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.040 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:20.958 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.043 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:20.960 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.045 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:20.962 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.079 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:20.996 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.092 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:21.010 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.107 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:20.978 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.115 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:20.986 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:45.139 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:21.010 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.929 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.044+0000 7f047033d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.044+0000 7f047033d640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.044+0000 7f047033d640 10 monclient: monmap: 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:49.044873+0000 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:49.044873+0000 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.045 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:49.046 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:49.046 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:49.046 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.044+0000 7f047033d640 1 Processor -- start 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.045+0000 7f047033d640 1 -- start start 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.045+0000 7f047033d640 10 monclient: init 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient(hunting): picked mon.noname-a con 0x7f04680a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient(hunting): picked mon.noname-b con 0x7f046813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient(hunting): picked mon.noname-c con 0x7f04681430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468142520 con 0x7f04680a98f0 2024-03-25T18:15:49.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b28a0 con 0x7f04681430c0 2024-03-25T18:15:49.048 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b2a30 con 0x7f046813f710 2024-03-25T18:15:49.048 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.048 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.046+0000 7f047033d640 10 monclient(hunting): authenticate will time out at 7658.307943s 2024-03-25T18:15:49.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.060+0000 7f046e8da640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f04681430c0 0x7f046814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:39640/0 (socket says 172.21.15.46:39640) 2024-03-25T18:15:49.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.060+0000 7f046e8da640 1 -- 172.21.15.46:0/2904066253 learned_addr learned my addr 172.21.15.46:0/2904066253 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137063028 0 0) 0x7f04680b2a30 con 0x7f046813f710 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:15:49.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.070+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0454003520 con 0x7f046813f710 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615309127 0 0) 0x7f04680b28a0 con 0x7f04681430c0 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:15:49.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.078+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04680b2a30 con 0x7f04681430c0 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2013587676 0 0) 0x7f0454003520 con 0x7f046813f710 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 >> v1:172.21.15.164:6789/0 conn(0x7f04681430c0 legacy=0x7f046814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 >> v1:172.21.15.46:6789/0 conn(0x7f04680a98f0 legacy=0x7f04680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 monclient: found mon.noname-b 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680b44c0 con 0x7f046813f710 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0458001690 con 0x7f046813f710 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f047033d640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f047033d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f046d0d7640 10 monclient: dump: 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.091 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.092 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:49.092 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:49.092 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:49.092 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.092 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.089+0000 7f047033d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04580017f0 con 0x7f046813f710 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 1 -- 172.21.15.46:0/2904066253 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0458001c40 con 0x7f046813f710 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:49.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f046d0d7640 10 monclient: dump: 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f047033d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:49.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.095+0000 7f047033d640 4 set_mon_vals no callback set 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 10 monclient: shutdown 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 1 -- 172.21.15.46:0/2904066253 >> v1:172.21.15.46:6790/0 conn(0x7f046813f710 legacy=0x7f04681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 1 -- 172.21.15.46:0/2904066253 shutdown_connections 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 1 -- 172.21.15.46:0/2904066253 >> 172.21.15.46:0/2904066253 conn(0x7f04680b7ea0 msgr2=0x7f04680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 1 -- 172.21.15.46:0/2904066253 shutdown_connections 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.096+0000 7f047033d640 1 -- 172.21.15.46:0/2904066253 wait complete. 2024-03-25T18:15:49.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.097+0000 7f047033d640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.097+0000 7f047033d640 10 monclient: monmap: 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 librados: starting msgr at 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 librados: starting objecter 2024-03-25T18:15:49.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 Processor -- start 2024-03-25T18:15:49.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 -- start start 2024-03-25T18:15:49.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 librados: setting wanted keys 2024-03-25T18:15:49.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 1 librados: calling monclient init 2024-03-25T18:15:49.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient: init 2024-03-25T18:15:49.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:49.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient(hunting): picked mon.noname-b con 0x7f04680a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:49.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient(hunting): picked mon.noname-c con 0x7f046813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:49.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.098+0000 7f047033d640 10 monclient(hunting): picked mon.noname-a con 0x7f04681430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:49.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.099+0000 7f047033d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681d7a10 con 0x7f04681430c0 2024-03-25T18:15:49.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.099+0000 7f047033d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681d76e0 con 0x7f04680a98f0 2024-03-25T18:15:49.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.099+0000 7f047033d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681d5210 con 0x7f046813f710 2024-03-25T18:15:49.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.099+0000 7f047033d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.099+0000 7f047033d640 10 monclient(hunting): authenticate will time out at 7658.360658s 2024-03-25T18:15:49.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.108+0000 7f046d8d8640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f046813f710 0x7f04681d4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:45584/0 (socket says 172.21.15.46:45584) 2024-03-25T18:15:49.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.108+0000 7f046d8d8640 1 -- 172.21.15.46:0/889011771 learned_addr learned my addr 172.21.15.46:0/889011771 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:49.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.112+0000 7f045effd640 10 client.?.objecter ms_handle_connect 0x7f046813f710 2024-03-25T18:15:49.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.112+0000 7f045effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.119+0000 7f045effd640 10 client.?.objecter ms_handle_connect 0x7f04680a98f0 2024-03-25T18:15:49.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.119+0000 7f045effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.122 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.121+0000 7f045effd640 10 client.?.objecter ms_handle_connect 0x7f04681430c0 2024-03-25T18:15:49.122 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.121+0000 7f045effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506986425 0 0) 0x7f04681d76e0 con 0x7f04680a98f0 2024-03-25T18:15:49.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:15:49.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.133+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044c004eb0 con 0x7f04680a98f0 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875022707 0 0) 0x7f04681d7a10 con 0x7f04681430c0 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:15:49.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.135+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04681d76e0 con 0x7f04681430c0 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509338813 0 0) 0x7f04681d5210 con 0x7f046813f710 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:15:49.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.137+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04681d7a10 con 0x7f046813f710 2024-03-25T18:15:49.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884715956 0 0) 0x7f04681d76e0 con 0x7f04681430c0 2024-03-25T18:15:49.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04681d5210 con 0x7f04681430c0 2024-03-25T18:15:49.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0460001b90 con 0x7f04681430c0 2024-03-25T18:15:49.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.153 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 10 monclient(hunting): dump: 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.152+0000 7f045effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502108447 0 0) 0x7f044c004eb0 con 0x7f04680a98f0 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04681d76e0 con 0x7f04680a98f0 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f04640018b0 con 0x7f04680a98f0 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.158+0000 7f045effd640 10 monclient(hunting): dump: 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.159 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:49.160 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3031143543 0 0) 0x7f04681d76e0 con 0x7f04680a98f0 2024-03-25T18:15:49.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 >> v1:172.21.15.46:6790/0 conn(0x7f046813f710 legacy=0x7f04681d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 >> v1:172.21.15.46:6789/0 conn(0x7f04681430c0 legacy=0x7f04681384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 1 monclient: found mon.b 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04681d41e0 con 0x7f04680a98f0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f045effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 10 monclient: _renew_subs 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04681d4450 con 0x7f04680a98f0 2024-03-25T18:15:49.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:49.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 10 monclient: _renew_subs 2024-03-25T18:15:49.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:49.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0468138dd0 con 0x7f04680a98f0 2024-03-25T18:15:49.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.167+0000 7f047033d640 1 librados: init done 2024-03-25T18:15:49.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.168+0000 7f047033d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.168+0000 7f047033d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.168+0000 7f047033d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:49.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.168+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04681d7870 con 0x7f04680a98f0 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04640035f0 con 0x7f04680a98f0 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0464003760 con 0x7f04680a98f0 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f045effd640 10 monclient: dump: 2024-03-25T18:15:49.451 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:49.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.449+0000 7f046c8d6640 4 set_mon_vals no callback set 2024-03-25T18:15:49.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.450+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f0464019730 con 0x7f04680a98f0 2024-03-25T18:15:49.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.451+0000 7f046d8d8640 10 monclient: get_auth_request con 0x7f044c06df50 auth_method 0 2024-03-25T18:15:49.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.451+0000 7f045effd640 10 client.4537.objecter ms_handle_connect 0x7f044c06df50 2024-03-25T18:15:49.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.456+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f04640533e0 con 0x7f04680a98f0 2024-03-25T18:15:49.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.456+0000 7f045effd640 10 client.4537.objecter ms_dispatch 0x7f0468000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:49.457 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.456+0000 7f045effd640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:49.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.456+0000 7f045effd640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:49.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.456+0000 7f045effd640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T18:15:49.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.457+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f046404ed20 con 0x7f04680a98f0 2024-03-25T18:15:49.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.458+0000 7f045effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.458+0000 7f045effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.684+0000 7f047033d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.684+0000 7f047033d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.684+0000 7f047033d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.684+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04681d45e0 con 0x7f04680a98f0 2024-03-25T18:15:49.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.707+0000 7f045effd640 1 -- 172.21.15.46:0/889011771 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f0464050e20 con 0x7f04680a98f0 2024-03-25T18:15:49.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.707+0000 7f045effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:49.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.707+0000 7f045effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:49.709 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:49.709 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 10 librados: watch_flush enter 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 10 librados: watch_flush exit 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 10 client.4537.objecter successfully canceled tick 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 >> v1:172.21.15.164:6800/1612084612 conn(0x7f044c06df50 legacy=0x7f044c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 10 monclient: shutdown 2024-03-25T18:15:49.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 >> v1:172.21.15.164:6789/0 conn(0x7f04680a98f0 legacy=0x7f04681d7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 shutdown_connections 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 >> 172.21.15.46:0/889011771 conn(0x7f04680b7ea0 msgr2=0x7f0468074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 shutdown_connections 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 -- 172.21.15.46:0/889011771 wait complete. 2024-03-25T18:15:49.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:49.710+0000 7f047033d640 1 librados: shutdown 2024-03-25T18:15:49.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:56.531 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.647+0000 7f5370a1e640 10 monclient: get_monmap_and_config 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.647+0000 7f5370a1e640 10 monclient: build_initial_monmap 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.647+0000 7f5370a1e640 10 monclient: monmap: 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:56.648 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:15:56.648290+0000 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:15:56.648290+0000 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:15:56.649 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.648+0000 7f5370a1e640 1 Processor -- start 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.648+0000 7f5370a1e640 1 -- start start 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.648+0000 7f5370a1e640 10 monclient: init 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-c con 0x7f536c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-a con 0x7f536c13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-b con 0x7f536c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c142520 con 0x7f536c13f710 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0b28a0 con 0x7f536c0a98f0 2024-03-25T18:15:56.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0b2a30 con 0x7f536c1430c0 2024-03-25T18:15:56.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:56.651 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.649+0000 7f5370a1e640 10 monclient(hunting): authenticate will time out at 7665.910898s 2024-03-25T18:15:56.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.652+0000 7f536a59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f536c0a98f0 0x7f536c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50574/0 (socket says 172.21.15.46:50574) 2024-03-25T18:15:56.653 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.652+0000 7f536a59c640 1 -- 172.21.15.46:0/4189838665 learned_addr learned my addr 172.21.15.46:0/4189838665 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290260864 0 0) 0x7f536c0b28a0 con 0x7f536c0a98f0 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:15:56.666 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.665+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354003520 con 0x7f536c0a98f0 2024-03-25T18:15:56.677 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899034286 0 0) 0x7f536c0b2a30 con 0x7f536c1430c0 2024-03-25T18:15:56.678 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:56.678 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:56.678 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:56.678 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:15:56.678 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.677+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c0b28a0 con 0x7f536c1430c0 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034611585 0 0) 0x7f536c142520 con 0x7f536c13f710 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:15:56.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.680+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c0b2a30 con 0x7f536c13f710 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4268149822 0 0) 0x7f5354003520 con 0x7f536c0a98f0 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 >> v1:172.21.15.46:6790/0 conn(0x7f536c1430c0 legacy=0x7f536c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 >> v1:172.21.15.46:6789/0 conn(0x7f536c13f710 legacy=0x7f536c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: _finish_hunting 0 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 monclient: found mon.noname-c 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0b44c0 con 0x7f536c0a98f0 2024-03-25T18:15:56.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: _finish_auth 0 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: _check_auth_tickets 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f535c001690 con 0x7f536c0a98f0 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f536959a640 10 monclient: dump: 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:56.686 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:56.687 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f5370a1e640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:15:56.687 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.684+0000 7f5370a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:56.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f535c0017f0 con 0x7f536c0a98f0 2024-03-25T18:15:56.692 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 1 -- 172.21.15.46:0/4189838665 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f535c001c00 con 0x7f536c0a98f0 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f536959a640 10 monclient: dump: 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f5370a1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:56.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.691+0000 7f5370a1e640 4 set_mon_vals no callback set 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 10 monclient: shutdown 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 1 -- 172.21.15.46:0/4189838665 >> v1:172.21.15.164:6789/0 conn(0x7f536c0a98f0 legacy=0x7f536c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 1 -- 172.21.15.46:0/4189838665 shutdown_connections 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 1 -- 172.21.15.46:0/4189838665 >> 172.21.15.46:0/4189838665 conn(0x7f536c0b7ea0 msgr2=0x7f536c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 1 -- 172.21.15.46:0/4189838665 shutdown_connections 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.692+0000 7f5370a1e640 1 -- 172.21.15.46:0/4189838665 wait complete. 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.693+0000 7f5370a1e640 10 monclient: build_initial_monmap 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.693+0000 7f5370a1e640 10 monclient: monmap: 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:56.695 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.693+0000 7f5370a1e640 1 librados: starting msgr at 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.693+0000 7f5370a1e640 1 librados: starting objecter 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 1 Processor -- start 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 1 -- start start 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 1 librados: setting wanted keys 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 1 librados: calling monclient init 2024-03-25T18:15:56.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 10 monclient: init 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-b con 0x7f536c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.694+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-c con 0x7f536c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 10 monclient(hunting): picked mon.noname-a con 0x7f536c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c137ab0 con 0x7f536c1430c0 2024-03-25T18:15:56.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c138bd0 con 0x7f536c0a98f0 2024-03-25T18:15:56.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c138e40 con 0x7f536c13f710 2024-03-25T18:15:56.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:56.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.695+0000 7f5370a1e640 10 monclient(hunting): authenticate will time out at 7665.956638s 2024-03-25T18:15:56.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.703+0000 7f536a59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f536c0a98f0 0x7f536c137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50600/0 (socket says 172.21.15.46:50600) 2024-03-25T18:15:56.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.703+0000 7f536a59c640 1 -- 172.21.15.46:0/168476651 learned_addr learned my addr 172.21.15.46:0/168476651 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:15:56.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.704+0000 7f535affd640 10 client.?.objecter ms_handle_connect 0x7f536c0a98f0 2024-03-25T18:15:56.704 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.704+0000 7f535affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:56.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.707+0000 7f535affd640 10 client.?.objecter ms_handle_connect 0x7f536c1430c0 2024-03-25T18:15:56.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.707+0000 7f535affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694774444 0 0) 0x7f536c138bd0 con 0x7f536c0a98f0 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:15:56.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.708+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c004eb0 con 0x7f536c0a98f0 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173633446 0 0) 0x7f534c004eb0 con 0x7f536c0a98f0 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f536c138bd0 con 0x7f536c0a98f0 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f535c001bc0 con 0x7f536c0a98f0 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 10 monclient(hunting): dump: 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:56.717 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:56.718 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:56.718 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:56.718 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:56.718 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.718 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.716+0000 7f535affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:56.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 client.?.objecter ms_handle_connect 0x7f536c13f710 2024-03-25T18:15:56.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:56.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3372533236 0 0) 0x7f536c138bd0 con 0x7f536c0a98f0 2024-03-25T18:15:56.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 >> v1:172.21.15.46:6790/0 conn(0x7f536c13f710 legacy=0x7f536c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 >> v1:172.21.15.46:6789/0 conn(0x7f536c1430c0 legacy=0x7f536c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 1 monclient: found mon.b 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c139120 con 0x7f536c0a98f0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f535affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 10 monclient: _renew_subs 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c139400 con 0x7f536c0a98f0 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:56.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 10 monclient: _renew_subs 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c0b7880 con 0x7f536c0a98f0 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.720+0000 7f5370a1e640 1 librados: init done 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.721+0000 7f5370a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.721+0000 7f5370a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.721+0000 7f5370a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:56.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.721+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c137810 con 0x7f536c0a98f0 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f535c004a90 con 0x7f536c0a98f0 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f535c001250 con 0x7f536c0a98f0 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f535affd640 10 monclient: dump: 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:15:56.996 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:15:56.997 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.995+0000 7f5368d99640 4 set_mon_vals no callback set 2024-03-25T18:15:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.996+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f535c004f60 con 0x7f536c0a98f0 2024-03-25T18:15:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.996+0000 7f5369d9b640 10 monclient: get_auth_request con 0x7f534c06dc00 auth_method 0 2024-03-25T18:15:56.998 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:56.997+0000 7f535affd640 10 client.4543.objecter ms_handle_connect 0x7f534c06dc00 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f535c0549f0 con 0x7f536c0a98f0 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 10 client.4543.objecter ms_dispatch 0x7f536c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f535c050330 con 0x7f536c0a98f0 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.003 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.002+0000 7f535affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:57.228 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.227+0000 7f5370a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.228 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.227+0000 7f5370a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.228 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.227+0000 7f5370a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:15:57.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.227+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f536c0627b0 con 0x7f536c0a98f0 2024-03-25T18:15:57.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.242+0000 7f535affd640 1 -- 172.21.15.46:0/168476651 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f535c052430 con 0x7f536c0a98f0 2024-03-25T18:15:57.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.243+0000 7f535affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.244 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.243+0000 7f535affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:57.244 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:15:57.244 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:57.246 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 10 librados: watch_flush enter 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 10 librados: watch_flush exit 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 10 client.4543.objecter successfully canceled tick 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 >> v1:172.21.15.164:6800/1612084612 conn(0x7f534c06dc00 legacy=0x7f534c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 10 monclient: shutdown 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 >> v1:172.21.15.164:6789/0 conn(0x7f536c0a98f0 legacy=0x7f536c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 shutdown_connections 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.245+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 >> 172.21.15.46:0/168476651 conn(0x7f536c0b7ea0 msgr2=0x7f536c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.246+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 shutdown_connections 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.246+0000 7f5370a1e640 1 -- 172.21.15.46:0/168476651 wait complete. 2024-03-25T18:15:57.247 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:15:57.246+0000 7f5370a1e640 1 librados: shutdown 2024-03-25T18:15:57.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:04.064 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:04.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.177+0000 7fb91c655640 10 monclient: get_monmap_and_config 2024-03-25T18:16:04.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.177+0000 7fb91c655640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.177+0000 7fb91c655640 10 monclient: monmap: 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:04.178615+0000 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:04.178615+0000 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:04.179 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.178+0000 7fb91c655640 1 Processor -- start 2024-03-25T18:16:04.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.178+0000 7fb91c655640 1 -- start start 2024-03-25T18:16:04.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.178+0000 7fb91c655640 10 monclient: init 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-c con 0x7fb9140a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-b con 0x7fb91413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-a con 0x7fb9141430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb914142520 con 0x7fb9141430c0 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140b28a0 con 0x7fb9140a98f0 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.179+0000 7fb91c655640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140b2a30 con 0x7fb91413f710 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.180+0000 7fb91c655640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.180+0000 7fb91c655640 10 monclient(hunting): authenticate will time out at 7673.441551s 2024-03-25T18:16:04.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.191+0000 7fb919bf0640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fb91413f710 0x7fb9141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:55288/0 (socket says 172.21.15.46:55288) 2024-03-25T18:16:04.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.191+0000 7fb919bf0640 1 -- 172.21.15.46:0/3849145382 learned_addr learned my addr 172.21.15.46:0/3849145382 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244464819 0 0) 0x7fb9140b2a30 con 0x7fb91413f710 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900003520 con 0x7fb91413f710 2024-03-25T18:16:04.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469012634 0 0) 0x7fb9140b28a0 con 0x7fb9140a98f0 2024-03-25T18:16:04.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:16:04.200 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.198+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9140b2a30 con 0x7fb9140a98f0 2024-03-25T18:16:04.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572394219 0 0) 0x7fb900003520 con 0x7fb91413f710 2024-03-25T18:16:04.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 >> v1:172.21.15.164:6789/0 conn(0x7fb9140a98f0 legacy=0x7fb9140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 >> v1:172.21.15.46:6789/0 conn(0x7fb9141430c0 legacy=0x7fb91414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 monclient: found mon.noname-b 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140b44c0 con 0x7fb91413f710 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb91c655640 5 monclient: authenticate success, global_id 4559 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb91c655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb904001690 con 0x7fb91413f710 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb9193ef640 10 monclient: dump: 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.208 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.209 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.206+0000 7fb91c655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9040017f0 con 0x7fb91413f710 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 1 -- 172.21.15.46:0/3849145382 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb904001c00 con 0x7fb91413f710 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb91c655640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb91c655640 4 set_mon_vals no callback set 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.219+0000 7fb9193ef640 10 monclient: dump: 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.220 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.221 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.221 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:04.221 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:04.221 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:04.221 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 10 monclient: shutdown 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 1 -- 172.21.15.46:0/3849145382 >> v1:172.21.15.46:6790/0 conn(0x7fb91413f710 legacy=0x7fb9141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 1 -- 172.21.15.46:0/3849145382 shutdown_connections 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 1 -- 172.21.15.46:0/3849145382 >> 172.21.15.46:0/3849145382 conn(0x7fb9140b7ea0 msgr2=0x7fb9140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 1 -- 172.21.15.46:0/3849145382 shutdown_connections 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.220+0000 7fb91c655640 1 -- 172.21.15.46:0/3849145382 wait complete. 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.221+0000 7fb91c655640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.221+0000 7fb91c655640 10 monclient: monmap: 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:04.222 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:04.223 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 librados: starting msgr at 2024-03-25T18:16:04.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 librados: starting objecter 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 Processor -- start 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 -- start start 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 librados: setting wanted keys 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 1 librados: calling monclient init 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.222+0000 7fb91c655640 10 monclient: init 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-c con 0x7fb9140a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-a con 0x7fb91413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient(hunting): picked mon.noname-b con 0x7fb9141430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91413be70 con 0x7fb91413f710 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb914131660 con 0x7fb9141430c0 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9141317f0 con 0x7fb9140a98f0 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.223+0000 7fb91c655640 10 monclient(hunting): authenticate will time out at 7673.484717s 2024-03-25T18:16:04.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.230+0000 7fb91a3f1640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fb9140a98f0 0x7fb914130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:55334/0 (socket says 172.21.15.46:55334) 2024-03-25T18:16:04.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.230+0000 7fb91a3f1640 1 -- 172.21.15.46:0/493869081 learned_addr learned my addr 172.21.15.46:0/493869081 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:04.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.231+0000 7fb90affd640 10 client.?.objecter ms_handle_connect 0x7fb9140a98f0 2024-03-25T18:16:04.231 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.231+0000 7fb90affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.233+0000 7fb90affd640 10 client.?.objecter ms_handle_connect 0x7fb9141430c0 2024-03-25T18:16:04.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.233+0000 7fb90affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410477560 0 0) 0x7fb9141317f0 con 0x7fb9140a98f0 2024-03-25T18:16:04.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.239 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:16:04.240 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.238+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8004eb0 con 0x7fb9140a98f0 2024-03-25T18:16:04.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280870542 0 0) 0x7fb914131660 con 0x7fb9141430c0 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.241+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9141317f0 con 0x7fb9141430c0 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.242+0000 7fb90affd640 10 client.?.objecter ms_handle_connect 0x7fb91413f710 2024-03-25T18:16:04.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.242+0000 7fb90affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331862786 0 0) 0x7fb8f8004eb0 con 0x7fb9140a98f0 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb914131660 con 0x7fb9140a98f0 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb910001a10 con 0x7fb9140a98f0 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 10 monclient(hunting): dump: 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.249 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:04.250 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:04.250 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:04.250 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.248+0000 7fb90affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542521099 0 0) 0x7fb91413be70 con 0x7fb91413f710 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:16:04.259 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8004eb0 con 0x7fb91413f710 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176277245 0 0) 0x7fb9141317f0 con 0x7fb9141430c0 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb91413be70 con 0x7fb9141430c0 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb90c001b00 con 0x7fb9141430c0 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.258+0000 7fb90affd640 10 monclient(hunting): dump: 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:04.260 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418755396 0 0) 0x7fb914131660 con 0x7fb9140a98f0 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 >> v1:172.21.15.164:6789/0 conn(0x7fb9141430c0 legacy=0x7fb9141e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 >> v1:172.21.15.46:6789/0 conn(0x7fb91413f710 legacy=0x7fb9141d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 1 monclient: found mon.c 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.264 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914131980 con 0x7fb9140a98f0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb90affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 10 monclient: _renew_subs 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb914135c60 con 0x7fb9140a98f0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 10 monclient: _renew_subs 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140b7770 con 0x7fb9140a98f0 2024-03-25T18:16:04.265 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.263+0000 7fb91c655640 1 librados: init done 2024-03-25T18:16:04.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.264+0000 7fb91c655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.264+0000 7fb91c655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.264+0000 7fb91c655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.264+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9141371d0 con 0x7fb9140a98f0 2024-03-25T18:16:04.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb910001c50 con 0x7fb9140a98f0 2024-03-25T18:16:04.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:04.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb910003a60 con 0x7fb9140a98f0 2024-03-25T18:16:04.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb90affd640 10 monclient: dump: 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:04.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.595+0000 7fb918bee640 4 set_mon_vals no callback set 2024-03-25T18:16:04.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.596+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb910003c80 con 0x7fb9140a98f0 2024-03-25T18:16:04.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.597+0000 7fb919bf0640 10 monclient: get_auth_request con 0x7fb8f806e040 auth_method 0 2024-03-25T18:16:04.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.597+0000 7fb90affd640 10 client.4562.objecter ms_handle_connect 0x7fb8f806e040 2024-03-25T18:16:04.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.603+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb9100548b0 con 0x7fb9140a98f0 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.603+0000 7fb90affd640 10 client.4562.objecter ms_dispatch 0x7fb914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.603+0000 7fb90affd640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.603+0000 7fb90affd640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.604+0000 7fb90affd640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.604+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9100503c0 con 0x7fb9140a98f0 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.604+0000 7fb90affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.604+0000 7fb90affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:04.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.824+0000 7fb91c655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.825+0000 7fb91c655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.825+0000 7fb91c655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:04.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.825+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb914074cc0 con 0x7fb9140a98f0 2024-03-25T18:16:04.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.846+0000 7fb90affd640 1 -- 172.21.15.46:0/493869081 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb9100522f0 con 0x7fb9140a98f0 2024-03-25T18:16:04.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.846+0000 7fb90affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.846+0000 7fb90affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:04.847 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:04.847 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 10 librados: watch_flush enter 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 10 librados: watch_flush exit 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 10 client.4562.objecter successfully canceled tick 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb8f806e040 legacy=0x7fb8f8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 10 monclient: shutdown 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 >> v1:172.21.15.46:6790/0 conn(0x7fb9140a98f0 legacy=0x7fb914130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 shutdown_connections 2024-03-25T18:16:04.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.848+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 >> 172.21.15.46:0/493869081 conn(0x7fb9140b7ea0 msgr2=0x7fb914074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:04.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.849+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 shutdown_connections 2024-03-25T18:16:04.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.849+0000 7fb91c655640 1 -- 172.21.15.46:0/493869081 wait complete. 2024-03-25T18:16:04.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:04.849+0000 7fb91c655640 1 librados: shutdown 2024-03-25T18:16:04.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:05.033 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:40.951 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.101 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:41.018 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.137 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:07:41.054 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.154 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:41.024 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.156 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:41.073 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.161 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:41.079 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.162 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:41.033 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.191 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:07:41.062 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.199 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:07:41.116 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:11.666 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:11.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.778+0000 7f7322982640 10 monclient: get_monmap_and_config 2024-03-25T18:16:11.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.778+0000 7f7322982640 10 monclient: build_initial_monmap 2024-03-25T18:16:11.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.779+0000 7f7322982640 10 monclient: monmap: 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:11.779672+0000 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:11.779672+0000 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:11.780 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.779+0000 7f7322982640 1 Processor -- start 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.779+0000 7f7322982640 1 -- start start 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.779+0000 7f7322982640 10 monclient: init 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:11.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient(hunting): picked mon.noname-c con 0x7f731c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient(hunting): picked mon.noname-b con 0x7f731c13f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient(hunting): picked mon.noname-a con 0x7f731c1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c142520 con 0x7f731c1430c0 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0b28a0 con 0x7f731c0a98f0 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0b2a30 con 0x7f731c13f710 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:11.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.780+0000 7f7322982640 10 monclient(hunting): authenticate will time out at 7681.042088s 2024-03-25T18:16:11.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.789+0000 7f731bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f731c0a98f0 0x7f731c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:41440/0 (socket says 172.21.15.46:41440) 2024-03-25T18:16:11.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.789+0000 7f731bfff640 1 -- 172.21.15.46:0/2653667862 learned_addr learned my addr 172.21.15.46:0/2653667862 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982750468 0 0) 0x7f731c142520 con 0x7f731c1430c0 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:16:11.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.805+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7304003520 con 0x7f731c1430c0 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719413460 0 0) 0x7f731c0b28a0 con 0x7f731c0a98f0 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:16:11.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.806+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c142520 con 0x7f731c0a98f0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410748807 0 0) 0x7f7304003520 con 0x7f731c1430c0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 >> v1:172.21.15.46:6790/0 conn(0x7f731c13f710 legacy=0x7f731c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 >> v1:172.21.15.164:6789/0 conn(0x7f731c0a98f0 legacy=0x7f731c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 monclient: found mon.noname-a 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0b44c0 con 0x7f731c1430c0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:11.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7314001a10 con 0x7f731c1430c0 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f731affd640 10 monclient: dump: 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:11.819 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f7322982640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:16:11.820 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.817+0000 7f7322982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7314001b70 con 0x7f731c1430c0 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 1 -- 172.21.15.46:0/2653667862 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7314001f80 con 0x7f731c1430c0 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f731affd640 10 monclient: dump: 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.836 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:11.837 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:11.837 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:11.837 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.837 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f7322982640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:11.837 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.835+0000 7f7322982640 4 set_mon_vals no callback set 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 10 monclient: shutdown 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 1 -- 172.21.15.46:0/2653667862 >> v1:172.21.15.46:6789/0 conn(0x7f731c1430c0 legacy=0x7f731c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 1 -- 172.21.15.46:0/2653667862 shutdown_connections 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 1 -- 172.21.15.46:0/2653667862 >> 172.21.15.46:0/2653667862 conn(0x7f731c0b7ea0 msgr2=0x7f731c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 1 -- 172.21.15.46:0/2653667862 shutdown_connections 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.836+0000 7f7322982640 1 -- 172.21.15.46:0/2653667862 wait complete. 2024-03-25T18:16:11.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 10 monclient: build_initial_monmap 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 10 monclient: monmap: 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 librados: starting msgr at 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 librados: starting objecter 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 Processor -- start 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 -- start start 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 librados: setting wanted keys 2024-03-25T18:16:11.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 1 librados: calling monclient init 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.837+0000 7f7322982640 10 monclient: init 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient(hunting): picked mon.noname-a con 0x7f731c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient(hunting): picked mon.noname-b con 0x7f731c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient(hunting): picked mon.noname-c con 0x7f731c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c139830 con 0x7f731c0a98f0 2024-03-25T18:16:11.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c1db7d0 con 0x7f731c13f710 2024-03-25T18:16:11.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c1db410 con 0x7f731c1430c0 2024-03-25T18:16:11.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:11.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.838+0000 7f7322982640 10 monclient(hunting): authenticate will time out at 7681.100105s 2024-03-25T18:16:11.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.851+0000 7f731bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f731c0a98f0 0x7f731c1dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38598/0 (socket says 172.21.15.46:38598) 2024-03-25T18:16:11.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.851+0000 7f731bfff640 1 -- 172.21.15.46:0/711854405 learned_addr learned my addr 172.21.15.46:0/711854405 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:11.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.852+0000 7f7318ff9640 10 client.?.objecter ms_handle_connect 0x7f731c0a98f0 2024-03-25T18:16:11.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.852+0000 7f7318ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.856+0000 7f7318ff9640 10 client.?.objecter ms_handle_connect 0x7f731c13f710 2024-03-25T18:16:11.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.856+0000 7f7318ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.856+0000 7f7318ff9640 10 client.?.objecter ms_handle_connect 0x7f731c1430c0 2024-03-25T18:16:11.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.856+0000 7f7318ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008370630 0 0) 0x7f731c1db7d0 con 0x7f731c13f710 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308004eb0 con 0x7f731c13f710 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401249130 0 0) 0x7f731c1db410 con 0x7f731c1430c0 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c1db7d0 con 0x7f731c1430c0 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648853339 0 0) 0x7f731c139830 con 0x7f731c0a98f0 2024-03-25T18:16:11.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:16:11.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.865+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c1db410 con 0x7f731c0a98f0 2024-03-25T18:16:11.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181539262 0 0) 0x7f7308004eb0 con 0x7f731c13f710 2024-03-25T18:16:11.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f731c139830 con 0x7f731c13f710 2024-03-25T18:16:11.877 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f73100018c0 con 0x7f731c13f710 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.878 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693402771 0 0) 0x7f731c1db410 con 0x7f731c0a98f0 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7308004eb0 con 0x7f731c0a98f0 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f730c0019b0 con 0x7f731c0a98f0 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.875+0000 7f7318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:11.879 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025188837 0 0) 0x7f731c1db7d0 con 0x7f731c1430c0 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f731c1db410 con 0x7f731c1430c0 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7314001e80 con 0x7f731c1430c0 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.876+0000 7f7318ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:11.880 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:11.881 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:11.881 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:11.881 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:11.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821102531 0 0) 0x7f731c139830 con 0x7f731c13f710 2024-03-25T18:16:11.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 >> v1:172.21.15.46:6790/0 conn(0x7f731c1430c0 legacy=0x7f731c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 >> v1:172.21.15.46:6789/0 conn(0x7f731c0a98f0 legacy=0x7f731c1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 1 monclient: found mon.b 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c1db5a0 con 0x7f731c13f710 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 10 monclient: _renew_subs 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:11.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c0b1480 con 0x7f731c13f710 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 10 monclient: _renew_subs 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c0b0930 con 0x7f731c13f710 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.884+0000 7f7322982640 1 librados: init done 2024-03-25T18:16:11.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.885+0000 7f7322982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:11.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.885+0000 7f7322982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:11.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.885+0000 7f7322982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:11.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.885+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c0627b0 con 0x7f731c13f710 2024-03-25T18:16:11.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.891+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73100035f0 con 0x7f731c13f710 2024-03-25T18:16:11.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.891+0000 7f7318ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:11.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:11.891+0000 7f731a7fc640 4 set_mon_vals no callback set 2024-03-25T18:16:12.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.162+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7310003900 con 0x7f731c13f710 2024-03-25T18:16:12.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.162+0000 7f7318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.162+0000 7f7318ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:12.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.162+0000 7f7318ff9640 10 monclient: dump: 2024-03-25T18:16:12.163 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:12.164 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:12.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.163+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f73100198d0 con 0x7f731c13f710 2024-03-25T18:16:12.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.164+0000 7f731bfff640 10 monclient: get_auth_request con 0x7f730806e060 auth_method 0 2024-03-25T18:16:12.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.164+0000 7f7318ff9640 10 client.4555.objecter ms_handle_connect 0x7f730806e060 2024-03-25T18:16:12.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.169+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7310053650 con 0x7f731c13f710 2024-03-25T18:16:12.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.169+0000 7f7318ff9640 10 client.4555.objecter ms_dispatch 0x7f731c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:12.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.169+0000 7f7318ff9640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:12.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.169+0000 7f7318ff9640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:12.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.169+0000 7f7318ff9640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T18:16:12.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.171+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f731004ef90 con 0x7f731c13f710 2024-03-25T18:16:12.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.171+0000 7f7318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.171+0000 7f7318ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:12.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.394+0000 7f7322982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:12.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.394+0000 7f7322982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:12.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.394+0000 7f7322982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:12.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.394+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f731c0b0c80 con 0x7f731c13f710 2024-03-25T18:16:12.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.488+0000 7f7318ff9640 1 -- 172.21.15.46:0/711854405 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7310051090 con 0x7f731c13f710 2024-03-25T18:16:12.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.488+0000 7f7318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:12.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.488+0000 7f7318ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:12.489 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:12.489 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 10 librados: watch_flush enter 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 10 librados: watch_flush exit 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 10 client.4555.objecter successfully canceled tick 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 >> v1:172.21.15.164:6800/1612084612 conn(0x7f730806e060 legacy=0x7f7308070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 10 monclient: shutdown 2024-03-25T18:16:12.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 >> v1:172.21.15.164:6789/0 conn(0x7f731c13f710 legacy=0x7f731c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 shutdown_connections 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 >> 172.21.15.46:0/711854405 conn(0x7f731c0b7ea0 msgr2=0x7f731c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 shutdown_connections 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 -- 172.21.15.46:0/711854405 wait complete. 2024-03-25T18:16:12.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:12.491+0000 7f7322982640 1 librados: shutdown 2024-03-25T18:16:12.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:19.310 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.423+0000 7f1f69723640 10 monclient: get_monmap_and_config 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.423+0000 7f1f69723640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.423+0000 7f1f69723640 10 monclient: monmap: 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.425 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:19.425345+0000 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:19.425345+0000 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:19.426 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.424+0000 7f1f69723640 1 Processor -- start 2024-03-25T18:16:19.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.424+0000 7f1f69723640 1 -- start start 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.424+0000 7f1f69723640 10 monclient: init 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-b con 0x7f1f640b4d60 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-a con 0x7f1f640b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-c con 0x7f1f6414be90 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640af940 con 0x7f1f640b1730 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640b0a70 con 0x7f1f6414be90 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640b0d50 con 0x7f1f640b4d60 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.428 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.425+0000 7f1f69723640 10 monclient(hunting): authenticate will time out at 7688.687974s 2024-03-25T18:16:19.434 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.433+0000 7f1f62ffd640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f1f640b4d60 0x7f1f6413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:46642/0 (socket says 172.21.15.46:46642) 2024-03-25T18:16:19.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.433+0000 7f1f62ffd640 1 -- 172.21.15.46:0/3308335240 learned_addr learned my addr 172.21.15.46:0/3308335240 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277627348 0 0) 0x7f1f640b0d50 con 0x7f1f640b4d60 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:16:19.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.447+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c003520 con 0x7f1f640b4d60 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118703543 0 0) 0x7f1f640b0a70 con 0x7f1f6414be90 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:16:19.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.460+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f640b0d50 con 0x7f1f6414be90 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502481285 0 0) 0x7f1f640af940 con 0x7f1f640b1730 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:16:19.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.466+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f640b0a70 con 0x7f1f640b1730 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944724067 0 0) 0x7f1f4c003520 con 0x7f1f640b4d60 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 >> v1:172.21.15.164:6789/0 conn(0x7f1f6414be90 legacy=0x7f1f6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 >> v1:172.21.15.46:6789/0 conn(0x7f1f640b1730 legacy=0x7f1f640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 monclient: found mon.noname-b 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640b1030 con 0x7f1f640b4d60 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f54001690 con 0x7f1f640b4d60 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f69723640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f69723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f61ffb640 10 monclient: dump: 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.476 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.477 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.473+0000 7f1f69723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f540017f0 con 0x7f1f640b4d60 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 1 -- 172.21.15.46:0/3308335240 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f54001c00 con 0x7f1f640b4d60 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f69723640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f69723640 4 set_mon_vals no callback set 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.501+0000 7f1f61ffb640 10 monclient: dump: 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.504 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.505 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.505 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.505 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.505 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.505 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 10 monclient: shutdown 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 1 -- 172.21.15.46:0/3308335240 >> v1:172.21.15.46:6790/0 conn(0x7f1f640b4d60 legacy=0x7f1f6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 1 -- 172.21.15.46:0/3308335240 shutdown_connections 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 1 -- 172.21.15.46:0/3308335240 >> 172.21.15.46:0/3308335240 conn(0x7f1f640617a0 msgr2=0x7f1f64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 1 -- 172.21.15.46:0/3308335240 shutdown_connections 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.502+0000 7f1f69723640 1 -- 172.21.15.46:0/3308335240 wait complete. 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.503+0000 7f1f69723640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.503+0000 7f1f69723640 10 monclient: monmap: 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:19.507 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.503+0000 7f1f69723640 1 librados: starting msgr at 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.503+0000 7f1f69723640 1 librados: starting objecter 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 1 Processor -- start 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 1 -- start start 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 1 librados: setting wanted keys 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 1 librados: calling monclient init 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 10 monclient: init 2024-03-25T18:16:19.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.504+0000 7f1f69723640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-b con 0x7f1f640b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-c con 0x7f1f640b4d60 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient(hunting): picked mon.noname-a con 0x7f1f6414be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f64136870 con 0x7f1f6414be90 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f64136350 con 0x7f1f640b1730 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f64136fa0 con 0x7f1f640b4d60 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.505+0000 7f1f69723640 10 monclient(hunting): authenticate will time out at 7688.767757s 2024-03-25T18:16:19.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.513+0000 7f1f62ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1f640b1730 0x7f1f6412fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:51958/0 (socket says 172.21.15.46:51958) 2024-03-25T18:16:19.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.513+0000 7f1f62ffd640 1 -- 172.21.15.46:0/1417651288 learned_addr learned my addr 172.21.15.46:0/1417651288 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:19.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.520+0000 7f1f4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1f640b1730 2024-03-25T18:16:19.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.520+0000 7f1f4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.521+0000 7f1f4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1f640b4d60 2024-03-25T18:16:19.522 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.521+0000 7f1f4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.522+0000 7f1f4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1f6414be90 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.522+0000 7f1f4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274262620 0 0) 0x7f1f64136fa0 con 0x7f1f640b4d60 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:16:19.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.526+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f50004eb0 con 0x7f1f640b4d60 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576435420 0 0) 0x7f1f64136350 con 0x7f1f640b1730 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:16:19.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.529+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64136fa0 con 0x7f1f640b1730 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111406893 0 0) 0x7f1f64136870 con 0x7f1f6414be90 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:16:19.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.531+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64136350 con 0x7f1f6414be90 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406694754 0 0) 0x7f1f64136fa0 con 0x7f1f640b1730 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f64136870 con 0x7f1f640b1730 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f54001b40 con 0x7f1f640b1730 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:19.542 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2199885243 0 0) 0x7f1f50004eb0 con 0x7f1f640b4d60 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f64136fa0 con 0x7f1f640b4d60 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f580018f0 con 0x7f1f640b4d60 2024-03-25T18:16:19.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.540+0000 7f1f4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.544 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.548 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423538639 0 0) 0x7f1f64136350 con 0x7f1f6414be90 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f50004eb0 con 0x7f1f6414be90 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f5c001bd0 con 0x7f1f6414be90 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.547+0000 7f1f4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.549 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.550 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.550 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536395656 0 0) 0x7f1f64136870 con 0x7f1f640b1730 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 >> v1:172.21.15.46:6790/0 conn(0x7f1f640b4d60 legacy=0x7f1f641dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 >> v1:172.21.15.46:6789/0 conn(0x7f1f6414be90 legacy=0x7f1f641e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 1 monclient: found mon.b 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f64138620 con 0x7f1f640b1730 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 10 monclient: _renew_subs 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f641324e0 con 0x7f1f640b1730 2024-03-25T18:16:19.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 10 monclient: _renew_subs 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f64135f10 con 0x7f1f640b1730 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.558+0000 7f1f69723640 1 librados: init done 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.559+0000 7f1f69723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.559+0000 7f1f69723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.559+0000 7f1f69723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:19.563 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.559+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f64132880 con 0x7f1f640b1730 2024-03-25T18:16:19.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f540011f0 con 0x7f1f640b1730 2024-03-25T18:16:19.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f54004c30 con 0x7f1f640b1730 2024-03-25T18:16:19.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f4b7fe640 10 monclient: dump: 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:19.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.840+0000 7f1f617fa640 4 set_mon_vals no callback set 2024-03-25T18:16:19.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.841+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f1f5401ac40 con 0x7f1f640b1730 2024-03-25T18:16:19.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.842+0000 7f1f627fc640 10 monclient: get_auth_request con 0x7f1f5006e010 auth_method 0 2024-03-25T18:16:19.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.842+0000 7f1f4b7fe640 10 client.4561.objecter ms_handle_connect 0x7f1f5006e010 2024-03-25T18:16:19.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.846+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f1f540548f0 con 0x7f1f640b1730 2024-03-25T18:16:19.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.846+0000 7f1f4b7fe640 10 client.4561.objecter ms_dispatch 0x7f1f64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:19.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.846+0000 7f1f4b7fe640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:19.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.846+0000 7f1f4b7fe640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:19.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.846+0000 7f1f4b7fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T18:16:19.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.847+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f54050230 con 0x7f1f640b1730 2024-03-25T18:16:19.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.847+0000 7f1f4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.849 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:19.847+0000 7f1f4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.087+0000 7f1f69723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.087+0000 7f1f69723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.087+0000 7f1f69723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:20.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.087+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f640627b0 con 0x7f1f640b1730 2024-03-25T18:16:20.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.285+0000 7f1f4b7fe640 1 -- 172.21.15.46:0/1417651288 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f1f54052330 con 0x7f1f640b1730 2024-03-25T18:16:20.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.285+0000 7f1f4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.287 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.285+0000 7f1f4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:20.288 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:20.288 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 10 librados: watch_flush enter 2024-03-25T18:16:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 10 librados: watch_flush exit 2024-03-25T18:16:20.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 10 client.4561.objecter successfully canceled tick 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 >> v1:172.21.15.164:6800/1612084612 conn(0x7f1f5006e010 legacy=0x7f1f500704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 10 monclient: shutdown 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 >> v1:172.21.15.164:6789/0 conn(0x7f1f640b1730 legacy=0x7f1f6412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 shutdown_connections 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 >> 172.21.15.46:0/1417651288 conn(0x7f1f640617a0 msgr2=0x7f1f640b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 shutdown_connections 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 -- 172.21.15.46:0/1417651288 wait complete. 2024-03-25T18:16:20.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:20.288+0000 7f1f69723640 1 librados: shutdown 2024-03-25T18:16:20.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:25.182 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:01.099 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.186 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:01.104 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.223 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:01.140 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.242 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:01.113 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.255 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:01.126 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.258 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:01.175 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.269 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:01.186 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.278 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:01.149 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:25.306 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:01.223 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:27.108 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:27.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.227+0000 7fe5aff5d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:27.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.227+0000 7fe5aff5d640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.229 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.227+0000 7fe5aff5d640 10 monclient: monmap: 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:27.229517+0000 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:27.229517+0000 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:27.230 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.228+0000 7fe5aff5d640 1 Processor -- start 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.228+0000 7fe5aff5d640 1 -- start start 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.228+0000 7fe5aff5d640 10 monclient: init 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a81430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8142520 con 0x7fe5a80a98f0 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80b28a0 con 0x7fe5a81430c0 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80b2a30 con 0x7fe5a813f710 2024-03-25T18:16:27.232 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.233 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.229+0000 7fe5aff5d640 10 monclient(hunting): authenticate will time out at 7696.492411s 2024-03-25T18:16:27.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.232+0000 7fe5ae4fa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe5a81430c0 0x7fe5a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:59838/0 (socket says 172.21.15.46:59838) 2024-03-25T18:16:27.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.232+0000 7fe5ae4fa640 1 -- 172.21.15.46:0/3169067804 learned_addr learned my addr 172.21.15.46:0/3169067804 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496630665 0 0) 0x7fe5a80b28a0 con 0x7fe5a81430c0 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:16:27.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.246+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594003520 con 0x7fe5a81430c0 2024-03-25T18:16:27.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.254+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525239487 0 0) 0x7fe5a80b2a30 con 0x7fe5a813f710 2024-03-25T18:16:27.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.255+0000 7fe5accf7640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.255+0000 7fe5accf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.255+0000 7fe5accf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.255+0000 7fe5accf7640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:16:27.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.255+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a80b28a0 con 0x7fe5a813f710 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132073174 0 0) 0x7fe594003520 con 0x7fe5a81430c0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 >> v1:172.21.15.46:6790/0 conn(0x7fe5a813f710 legacy=0x7fe5a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 >> v1:172.21.15.46:6789/0 conn(0x7fe5a80a98f0 legacy=0x7fe5a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 monclient: found mon.noname-c 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80b44c0 con 0x7fe5a81430c0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5a0001a10 con 0x7fe5a81430c0 2024-03-25T18:16:27.262 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5aff5d640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5aff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5accf7640 10 monclient: dump: 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.263 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.259+0000 7fe5aff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5a0001b70 con 0x7fe5a81430c0 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 1 -- 172.21.15.46:0/3169067804 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5a0001f80 con 0x7fe5a81430c0 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5accf7640 10 monclient: dump: 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.280 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5aff5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:27.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.278+0000 7fe5aff5d640 4 set_mon_vals no callback set 2024-03-25T18:16:27.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 10 monclient: shutdown 2024-03-25T18:16:27.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:27.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 1 -- 172.21.15.46:0/3169067804 >> v1:172.21.15.164:6789/0 conn(0x7fe5a81430c0 legacy=0x7fe5a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.283 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 1 -- 172.21.15.46:0/3169067804 shutdown_connections 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 1 -- 172.21.15.46:0/3169067804 >> 172.21.15.46:0/3169067804 conn(0x7fe5a80b7ea0 msgr2=0x7fe5a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 1 -- 172.21.15.46:0/3169067804 shutdown_connections 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.279+0000 7fe5aff5d640 1 -- 172.21.15.46:0/3169067804 wait complete. 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 10 monclient: monmap: 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:27.284 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 librados: starting msgr at 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 librados: starting objecter 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 Processor -- start 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 -- start start 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 librados: setting wanted keys 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 1 librados: calling monclient init 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.280+0000 7fe5aff5d640 10 monclient: init 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a813f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a81430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:27.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a81373f0 con 0x7fe5a81430c0 2024-03-25T18:16:27.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a81db280 con 0x7fe5a80a98f0 2024-03-25T18:16:27.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8138f40 con 0x7fe5a813f710 2024-03-25T18:16:27.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.281+0000 7fe5aff5d640 10 monclient(hunting): authenticate will time out at 7696.543647s 2024-03-25T18:16:27.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.291+0000 7fe5ad4f8640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fe5a813f710 0x7fe5a8132bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:39784/0 (socket says 172.21.15.46:39784) 2024-03-25T18:16:27.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.291+0000 7fe5ad4f8640 1 -- 172.21.15.46:0/20704449 learned_addr learned my addr 172.21.15.46:0/20704449 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:27.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.295+0000 7fe59e7fc640 10 client.?.objecter ms_handle_connect 0x7fe5a81430c0 2024-03-25T18:16:27.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.295+0000 7fe59e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.296+0000 7fe59e7fc640 10 client.?.objecter ms_handle_connect 0x7fe5a813f710 2024-03-25T18:16:27.298 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.296+0000 7fe59e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.304+0000 7fe59e7fc640 10 client.?.objecter ms_handle_connect 0x7fe5a80a98f0 2024-03-25T18:16:27.306 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.304+0000 7fe59e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815970045 0 0) 0x7fe5a81373f0 con 0x7fe5a81430c0 2024-03-25T18:16:27.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:16:27.309 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.307+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c004eb0 con 0x7fe5a81430c0 2024-03-25T18:16:27.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792173471 0 0) 0x7fe5a81db280 con 0x7fe5a80a98f0 2024-03-25T18:16:27.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:16:27.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.313+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a81373f0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2253302664 0 0) 0x7fe58c004eb0 con 0x7fe5a81430c0 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a81db280 con 0x7fe5a81430c0 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5a0001ec0 con 0x7fe5a81430c0 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:27.316 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.314+0000 7fe59e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716453588 0 0) 0x7fe5a8138f40 con 0x7fe5a813f710 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:16:27.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.316+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c004eb0 con 0x7fe5a813f710 2024-03-25T18:16:27.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.320+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 639611778 0 0) 0x7fe5a81373f0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.320+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a8138f40 con 0x7fe5a80a98f0 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.324+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5a40018b0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.324+0000 7fe59e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.324+0000 7fe59e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.324+0000 7fe59e7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.326 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152737953 0 0) 0x7fe58c004eb0 con 0x7fe5a813f710 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a81373f0 con 0x7fe5a813f710 2024-03-25T18:16:27.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5980019b0 con 0x7fe5a813f710 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.325+0000 7fe59e7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.328 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2981527969 0 0) 0x7fe5a8138f40 con 0x7fe5a80a98f0 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 >> v1:172.21.15.46:6790/0 conn(0x7fe5a813f710 legacy=0x7fe5a8132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 >> v1:172.21.15.46:6789/0 conn(0x7fe5a81430c0 legacy=0x7fe5a81332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 1 monclient: found mon.b 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a81339d0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe59e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 10 monclient: _renew_subs 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a8137f50 con 0x7fe5a80a98f0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 10 monclient: _renew_subs 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a813c550 con 0x7fe5a80a98f0 2024-03-25T18:16:27.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.329+0000 7fe5aff5d640 1 librados: init done 2024-03-25T18:16:27.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.330+0000 7fe5aff5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.330+0000 7fe5aff5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.330+0000 7fe5aff5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.333 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.330+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a80b3ef0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5a40035f0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe5a4001d80 con 0x7fe5a80a98f0 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:27.612 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 10 monclient: dump: 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59ffff640 4 set_mon_vals no callback set 2024-03-25T18:16:27.613 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.610+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fe5a40197b0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.611+0000 7fe5ad4f8640 10 monclient: get_auth_request con 0x7fe58c06e110 auth_method 0 2024-03-25T18:16:27.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.611+0000 7fe59e7fc640 10 client.4567.objecter ms_handle_connect 0x7fe58c06e110 2024-03-25T18:16:27.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fe5a4053520 con 0x7fe5a80a98f0 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 10 client.4567.objecter ms_dispatch 0x7fe5a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5a404ee60 con 0x7fe5a80a98f0 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.616+0000 7fe59e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.832+0000 7fe5aff5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.832+0000 7fe5aff5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.832+0000 7fe5aff5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.832+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a80627b0 con 0x7fe5a80a98f0 2024-03-25T18:16:27.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.852+0000 7fe59e7fc640 1 -- 172.21.15.46:0/20704449 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fe5a4050f60 con 0x7fe5a80a98f0 2024-03-25T18:16:27.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.852+0000 7fe59e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.852+0000 7fe59e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:27.854 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:27.854 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 10 librados: watch_flush enter 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 10 librados: watch_flush exit 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 10 client.4567.objecter successfully canceled tick 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 >> v1:172.21.15.164:6800/1612084612 conn(0x7fe58c06e110 legacy=0x7fe58c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.854+0000 7fe5aff5d640 10 monclient: shutdown 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 >> v1:172.21.15.164:6789/0 conn(0x7fe5a80a98f0 legacy=0x7fe5a81dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 shutdown_connections 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 >> 172.21.15.46:0/20704449 conn(0x7fe5a80b7ea0 msgr2=0x7fe5a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 shutdown_connections 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 -- 172.21.15.46:0/20704449 wait complete. 2024-03-25T18:16:27.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:27.855+0000 7fe5aff5d640 1 librados: shutdown 2024-03-25T18:16:27.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:34.676 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:34.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.790+0000 7f2bc8572640 10 monclient: get_monmap_and_config 2024-03-25T18:16:34.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.790+0000 7f2bc8572640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.790+0000 7f2bc8572640 10 monclient: monmap: 2024-03-25T18:16:34.792 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:34.792434+0000 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:34.792434+0000 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:34.793 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.791+0000 7f2bc8572640 1 Processor -- start 2024-03-25T18:16:34.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.791+0000 7f2bc8572640 1 -- start start 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.791+0000 7f2bc8572640 10 monclient: init 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc00a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc01430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc0142520 con 0x7f2bc01430c0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc00b28a0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc00b2a30 con 0x7f2bc013f710 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.792+0000 7f2bc8572640 10 monclient(hunting): authenticate will time out at 7704.055199s 2024-03-25T18:16:34.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.803+0000 7f2bc5b0d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f2bc013f710 0x7f2bc01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:53116/0 (socket says 172.21.15.46:53116) 2024-03-25T18:16:34.805 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.803+0000 7f2bc5b0d640 1 -- 172.21.15.46:0/3900241991 learned_addr learned my addr 172.21.15.46:0/3900241991 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548690864 0 0) 0x7f2bc00b28a0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:16:34.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.814+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bac003520 con 0x7f2bc00a98f0 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672299301 0 0) 0x7f2bc00b2a30 con 0x7f2bc013f710 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:16:34.817 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.815+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc00b28a0 con 0x7f2bc013f710 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280406962 0 0) 0x7f2bac003520 con 0x7f2bc00a98f0 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 >> v1:172.21.15.46:6790/0 conn(0x7f2bc013f710 legacy=0x7f2bc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 >> v1:172.21.15.46:6789/0 conn(0x7f2bc01430c0 legacy=0x7f2bc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 monclient: found mon.noname-c 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc00b44c0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bbc001690 con 0x7f2bc00a98f0 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc530c640 10 monclient: dump: 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:34.826 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:34.827 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc8572640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:16:34.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.823+0000 7f2bc8572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bbc0017f0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 1 -- 172.21.15.46:0/3900241991 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bbc001c00 con 0x7f2bc00a98f0 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc8572640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc8572640 4 set_mon_vals no callback set 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc530c640 10 monclient: dump: 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.842 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.843 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.843 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:34.843 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:34.843 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:34.843 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc8572640 10 monclient: shutdown 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc8572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.840+0000 7f2bc8572640 1 -- 172.21.15.46:0/3900241991 >> v1:172.21.15.164:6789/0 conn(0x7f2bc00a98f0 legacy=0x7f2bc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 1 -- 172.21.15.46:0/3900241991 shutdown_connections 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 1 -- 172.21.15.46:0/3900241991 >> 172.21.15.46:0/3900241991 conn(0x7f2bc00b7ea0 msgr2=0x7f2bc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:34.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 1 -- 172.21.15.46:0/3900241991 shutdown_connections 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 1 -- 172.21.15.46:0/3900241991 wait complete. 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.841+0000 7f2bc8572640 10 monclient: monmap: 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 librados: starting msgr at 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 librados: starting objecter 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 Processor -- start 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 -- start start 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 librados: setting wanted keys 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 1 librados: calling monclient init 2024-03-25T18:16:34.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.842+0000 7f2bc8572640 10 monclient: init 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc00a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc01430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc0139a40 con 0x7f2bc013f710 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc0133f20 con 0x7f2bc01430c0 2024-03-25T18:16:34.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc00ab950 con 0x7f2bc00a98f0 2024-03-25T18:16:34.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.843+0000 7f2bc8572640 10 monclient(hunting): authenticate will time out at 7704.106234s 2024-03-25T18:16:34.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.857+0000 7f2bc5b0d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2bc013f710 0x7f2bc0133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52196/0 (socket says 172.21.15.46:52196) 2024-03-25T18:16:34.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.857+0000 7f2bc5b0d640 1 -- 172.21.15.46:0/2986723964 learned_addr learned my addr 172.21.15.46:0/2986723964 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:34.862 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.860+0000 7f2bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f2bc013f710 2024-03-25T18:16:34.862 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.861+0000 7f2bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.866+0000 7f2bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f2bc01430c0 2024-03-25T18:16:34.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.866+0000 7f2bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.866+0000 7f2bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f2bc00a98f0 2024-03-25T18:16:34.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.866+0000 7f2bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658056087 0 0) 0x7f2bc0139a40 con 0x7f2bc013f710 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:16:34.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.879+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba4004eb0 con 0x7f2bc013f710 2024-03-25T18:16:34.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.884+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383355155 0 0) 0x7f2bc0133f20 con 0x7f2bc01430c0 2024-03-25T18:16:34.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc0139a40 con 0x7f2bc01430c0 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787807401 0 0) 0x7f2bc00ab950 con 0x7f2bc00a98f0 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:16:34.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.885+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc0133f20 con 0x7f2bc00a98f0 2024-03-25T18:16:34.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069631614 0 0) 0x7f2ba4004eb0 con 0x7f2bc013f710 2024-03-25T18:16:34.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc00ab950 con 0x7f2bc013f710 2024-03-25T18:16:34.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bb00019f0 con 0x7f2bc013f710 2024-03-25T18:16:34.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.890+0000 7f2bb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:34.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.900+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1391713344 0 0) 0x7f2bc0133f20 con 0x7f2bc00a98f0 2024-03-25T18:16:34.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.901+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba4004eb0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.902 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.901+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bbc001b00 con 0x7f2bc00a98f0 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.901+0000 7f2bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.901+0000 7f2bb6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.901+0000 7f2bb6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:34.903 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288630306 0 0) 0x7f2bc0139a40 con 0x7f2bc01430c0 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc0133f20 con 0x7f2bc01430c0 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bb8001b10 con 0x7f2bc01430c0 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.905+0000 7f2bb6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:34.907 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2682215699 0 0) 0x7f2ba4004eb0 con 0x7f2bc00a98f0 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 >> v1:172.21.15.164:6789/0 conn(0x7f2bc01430c0 legacy=0x7f2bc0133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 >> v1:172.21.15.46:6789/0 conn(0x7f2bc013f710 legacy=0x7f2bc0133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 1 monclient: found mon.c 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc00abc30 con 0x7f2bc00a98f0 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bc8572640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bc8572640 10 monclient: _renew_subs 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bc8572640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.907+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc00abf10 con 0x7f2bc00a98f0 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.908+0000 7f2bc8572640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.908+0000 7f2bc8572640 10 monclient: _renew_subs 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.908+0000 7f2bc8572640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.908+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc0139370 con 0x7f2bc00a98f0 2024-03-25T18:16:34.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.908+0000 7f2bc8572640 1 librados: init done 2024-03-25T18:16:34.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.909+0000 7f2bc8572640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.909+0000 7f2bc8572640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.909+0000 7f2bc8572640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:34.912 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:34.909+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc00627b0 con 0x7f2bc00a98f0 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bbc001c60 con 0x7f2bc00a98f0 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2bbc003a40 con 0x7f2bc00a98f0 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bb6ffd640 10 monclient: dump: 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:35.234 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:35.235 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.232+0000 7f2bc4b0b640 4 set_mon_vals no callback set 2024-03-25T18:16:35.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.233+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f2bbc019a10 con 0x7f2bc00a98f0 2024-03-25T18:16:35.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.233+0000 7f2bc5b0d640 10 monclient: get_auth_request con 0x7f2ba406e100 auth_method 0 2024-03-25T18:16:35.236 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.234+0000 7f2bb6ffd640 10 client.4586.objecter ms_handle_connect 0x7f2ba406e100 2024-03-25T18:16:35.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f2bbc0535c0 con 0x7f2bc00a98f0 2024-03-25T18:16:35.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 10 client.4586.objecter ms_dispatch 0x7f2bc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:35.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:35.241 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:35.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T18:16:35.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bbc04f0d0 con 0x7f2bc00a98f0 2024-03-25T18:16:35.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.239+0000 7f2bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:35.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.487+0000 7f2bc8572640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.487+0000 7f2bc8572640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.487+0000 7f2bc8572640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:16:35.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.487+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bc0138980 con 0x7f2bc00a98f0 2024-03-25T18:16:35.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.523+0000 7f2bb6ffd640 1 -- 172.21.15.46:0/2986723964 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f2bc0138980 con 0x7f2bc00a98f0 2024-03-25T18:16:35.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.523+0000 7f2bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.523+0000 7f2bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:35.525 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:35.525 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 10 librados: watch_flush enter 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 10 librados: watch_flush exit 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 10 client.4586.objecter successfully canceled tick 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 >> v1:172.21.15.164:6800/1612084612 conn(0x7f2ba406e100 legacy=0x7f2ba40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 10 monclient: shutdown 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.525+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 >> v1:172.21.15.46:6790/0 conn(0x7f2bc00a98f0 legacy=0x7f2bc0138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.526+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 shutdown_connections 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.526+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 >> 172.21.15.46:0/2986723964 conn(0x7f2bc00b7ea0 msgr2=0x7f2bc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.526+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 shutdown_connections 2024-03-25T18:16:35.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.526+0000 7f2bc8572640 1 -- 172.21.15.46:0/2986723964 wait complete. 2024-03-25T18:16:35.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:35.526+0000 7f2bc8572640 1 librados: shutdown 2024-03-25T18:16:35.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:42.347 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:42.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.459+0000 7fba58b54640 10 monclient: get_monmap_and_config 2024-03-25T18:16:42.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.459+0000 7fba58b54640 10 monclient: build_initial_monmap 2024-03-25T18:16:42.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.459+0000 7fba58b54640 10 monclient: monmap: 2024-03-25T18:16:42.461 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:42.461 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:42.461164+0000 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:42.461164+0000 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:42.462 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.460+0000 7fba58b54640 1 Processor -- start 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.460+0000 7fba58b54640 1 -- start start 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.460+0000 7fba58b54640 10 monclient: init 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-b con 0x7fba54149f50 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-a con 0x7fba5406c780 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-c con 0x7fba540b1b10 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:42.464 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5414a330 con 0x7fba5406c780 2024-03-25T18:16:42.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba54070660 con 0x7fba540b1b10 2024-03-25T18:16:42.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5406d7c0 con 0x7fba54149f50 2024-03-25T18:16:42.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:42.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.461+0000 7fba58b54640 10 monclient(hunting): authenticate will time out at 7711.724175s 2024-03-25T18:16:42.478 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.477+0000 7fba5259c640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fba54149f50 0x7fba540b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:57692/0 (socket says 172.21.15.46:57692) 2024-03-25T18:16:42.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.477+0000 7fba5259c640 1 -- 172.21.15.46:0/2368987836 learned_addr learned my addr 172.21.15.46:0/2368987836 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:42.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375413011 0 0) 0x7fba5406d7c0 con 0x7fba54149f50 2024-03-25T18:16:42.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:16:42.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.487+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c003520 con 0x7fba54149f50 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762156597 0 0) 0x7fba5414a330 con 0x7fba5406c780 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:16:42.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.498+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5406d7c0 con 0x7fba5406c780 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590976595 0 0) 0x7fba54070660 con 0x7fba540b1b10 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:16:42.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.503+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5414a330 con 0x7fba540b1b10 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2250807277 0 0) 0x7fba3c003520 con 0x7fba54149f50 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 >> v1:172.21.15.164:6789/0 conn(0x7fba540b1b10 legacy=0x7fba54152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 >> v1:172.21.15.46:6789/0 conn(0x7fba5406c780 legacy=0x7fba540b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 monclient: found mon.noname-b 2024-03-25T18:16:42.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5406d3e0 con 0x7fba54149f50 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: _finish_auth 0 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba44001690 con 0x7fba54149f50 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba58b54640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba58b54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba5159a640 10 monclient: dump: 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.519 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.516+0000 7fba58b54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba440017f0 con 0x7fba54149f50 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 1 -- 172.21.15.46:0/2368987836 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba44001c00 con 0x7fba54149f50 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba5159a640 10 monclient: dump: 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.528 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba58b54640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:42.529 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.526+0000 7fba58b54640 4 set_mon_vals no callback set 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 10 monclient: shutdown 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 1 -- 172.21.15.46:0/2368987836 >> v1:172.21.15.46:6790/0 conn(0x7fba54149f50 legacy=0x7fba540b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 1 -- 172.21.15.46:0/2368987836 shutdown_connections 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 1 -- 172.21.15.46:0/2368987836 >> 172.21.15.46:0/2368987836 conn(0x7fba540b5d10 msgr2=0x7fba540b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 1 -- 172.21.15.46:0/2368987836 shutdown_connections 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.527+0000 7fba58b54640 1 -- 172.21.15.46:0/2368987836 wait complete. 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.528+0000 7fba58b54640 10 monclient: build_initial_monmap 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.528+0000 7fba58b54640 10 monclient: monmap: 2024-03-25T18:16:42.531 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 librados: starting msgr at 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 librados: starting objecter 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 Processor -- start 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 -- start start 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 librados: setting wanted keys 2024-03-25T18:16:42.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 librados: calling monclient init 2024-03-25T18:16:42.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient: init 2024-03-25T18:16:42.533 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-a con 0x7fba5406c780 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-c con 0x7fba540b1b10 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient(hunting): picked mon.noname-b con 0x7fba54149f50 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba541df6e0 con 0x7fba5406c780 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540acfe0 con 0x7fba54149f50 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba541df170 con 0x7fba540b1b10 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:42.534 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.529+0000 7fba58b54640 10 monclient(hunting): authenticate will time out at 7711.792517s 2024-03-25T18:16:42.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.540+0000 7fba51d9b640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fba540b1b10 0x7fba540ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:57748/0 (socket says 172.21.15.46:57748) 2024-03-25T18:16:42.542 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.540+0000 7fba51d9b640 1 -- 172.21.15.46:0/3357080889 learned_addr learned my addr 172.21.15.46:0/3357080889 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:42.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.541+0000 7fba42ffd640 10 client.?.objecter ms_handle_connect 0x7fba5406c780 2024-03-25T18:16:42.543 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.541+0000 7fba42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:42.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.542+0000 7fba42ffd640 10 client.?.objecter ms_handle_connect 0x7fba54149f50 2024-03-25T18:16:42.544 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.542+0000 7fba42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:42.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.543+0000 7fba42ffd640 10 client.?.objecter ms_handle_connect 0x7fba540b1b10 2024-03-25T18:16:42.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.543+0000 7fba42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:42.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957624455 0 0) 0x7fba541df170 con 0x7fba540b1b10 2024-03-25T18:16:42.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:16:42.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.554+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba34004eb0 con 0x7fba540b1b10 2024-03-25T18:16:42.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076640955 0 0) 0x7fba541df6e0 con 0x7fba5406c780 2024-03-25T18:16:42.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.559 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:16:42.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.557+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba541df170 con 0x7fba5406c780 2024-03-25T18:16:42.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328704572 0 0) 0x7fba540acfe0 con 0x7fba54149f50 2024-03-25T18:16:42.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:42.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:42.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:42.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:16:42.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.560+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba541df6e0 con 0x7fba54149f50 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438137696 0 0) 0x7fba541df170 con 0x7fba5406c780 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba540acfe0 con 0x7fba5406c780 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba44001b00 con 0x7fba5406c780 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:42.570 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.571 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.568+0000 7fba42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031070223 0 0) 0x7fba34004eb0 con 0x7fba540b1b10 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba541df170 con 0x7fba540b1b10 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba48001a30 con 0x7fba540b1b10 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.569+0000 7fba42ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.572 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:42.573 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3017043727 0 0) 0x7fba540acfe0 con 0x7fba5406c780 2024-03-25T18:16:42.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 >> v1:172.21.15.46:6790/0 conn(0x7fba540b1b10 legacy=0x7fba540ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 >> v1:172.21.15.164:6789/0 conn(0x7fba54149f50 legacy=0x7fba541e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 1 monclient: found mon.a 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba541df300 con 0x7fba5406c780 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 10 monclient: _renew_subs 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba540af7c0 con 0x7fba5406c780 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:42.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 10 monclient: _renew_subs 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba54146d10 con 0x7fba5406c780 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.577+0000 7fba58b54640 1 librados: init done 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.578+0000 7fba58b54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.578+0000 7fba58b54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:42.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.578+0000 7fba58b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:16:42.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.578+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba540b15e0 con 0x7fba5406c780 2024-03-25T18:16:42.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba44001ca0 con 0x7fba5406c780 2024-03-25T18:16:42.603 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fba44004c30 con 0x7fba5406c780 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba42ffd640 10 monclient: dump: 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:42.604 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:42.605 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:42.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.601+0000 7fba50d99640 4 set_mon_vals no callback set 2024-03-25T18:16:42.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.602+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fba4401ac00 con 0x7fba5406c780 2024-03-25T18:16:42.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.603+0000 7fba51d9b640 10 monclient: get_auth_request con 0x7fba3406df20 auth_method 0 2024-03-25T18:16:42.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.603+0000 7fba42ffd640 10 client.4575.objecter ms_handle_connect 0x7fba3406df20 2024-03-25T18:16:42.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.607+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fba44054780 con 0x7fba5406c780 2024-03-25T18:16:42.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.607+0000 7fba42ffd640 10 client.4575.objecter ms_dispatch 0x7fba54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:42.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.607+0000 7fba42ffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:42.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.607+0000 7fba42ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:42.609 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.607+0000 7fba42ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:16:42.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.894+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba440500c0 con 0x7fba5406c780 2024-03-25T18:16:42.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.894+0000 7fba42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:42.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:42.894+0000 7fba42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.146+0000 7fba58b54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.146+0000 7fba58b54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.146+0000 7fba58b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.146+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba540ad500 con 0x7fba5406c780 2024-03-25T18:16:43.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.193+0000 7fba42ffd640 1 -- 172.21.15.46:0/3357080889 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fba540ad500 con 0x7fba5406c780 2024-03-25T18:16:43.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.193+0000 7fba42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.193+0000 7fba42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:43.195 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:43.195 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 10 librados: watch_flush enter 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 10 librados: watch_flush exit 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 >> v1:172.21.15.164:6800/1612084612 conn(0x7fba3406df20 legacy=0x7fba340703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 10 monclient: shutdown 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.198 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 >> v1:172.21.15.46:6789/0 conn(0x7fba5406c780 legacy=0x7fba540aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 shutdown_connections 2024-03-25T18:16:43.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.195+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 >> 172.21.15.46:0/3357080889 conn(0x7fba540b5d10 msgr2=0x7fba540b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.196+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 shutdown_connections 2024-03-25T18:16:43.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.196+0000 7fba58b54640 1 -- 172.21.15.46:0/3357080889 wait complete. 2024-03-25T18:16:43.199 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:43.196+0000 7fba58b54640 1 librados: shutdown 2024-03-25T18:16:43.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:45.501 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:21.418 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.518 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:21.435 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.552 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:21.470 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.566 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:21.437 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.571 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:21.489 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.576 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:21.447 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.586 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:21.504 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.612 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:21.483 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:45.615 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:21.533 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:50.016 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.132+0000 7ff4f5f1c640 10 monclient: get_monmap_and_config 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.132+0000 7ff4f5f1c640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.132+0000 7ff4f5f1c640 10 monclient: monmap: 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:50.134 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:50.134303+0000 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:50.134303+0000 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:50.135 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.133+0000 7ff4f5f1c640 1 Processor -- start 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.133+0000 7ff4f5f1c640 1 -- start start 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.133+0000 7ff4f5f1c640 10 monclient: init 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f00a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:50.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f01430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f0142520 con 0x7ff4f013f710 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00b28a0 con 0x7ff4f01430c0 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00b2a30 con 0x7ff4f00a98f0 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.134+0000 7ff4f5f1c640 10 monclient(hunting): authenticate will time out at 7719.396725s 2024-03-25T18:16:50.145 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.144+0000 7ff4ef7fe640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7ff4f00a98f0 0x7ff4f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:43356/0 (socket says 172.21.15.46:43356) 2024-03-25T18:16:50.145 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.144+0000 7ff4ef7fe640 1 -- 172.21.15.46:0/3944971589 learned_addr learned my addr 172.21.15.46:0/3944971589 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:50.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.148+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722859422 0 0) 0x7ff4f00b2a30 con 0x7ff4f00a98f0 2024-03-25T18:16:50.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.149+0000 7ff4ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.150 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.149+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.151 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.149+0000 7ff4ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.151 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.149+0000 7ff4ee7fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:16:50.151 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.149+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d8003520 con 0x7ff4f00a98f0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2416445832 0 0) 0x7ff4d8003520 con 0x7ff4f00a98f0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 >> v1:172.21.15.164:6789/0 conn(0x7ff4f01430c0 legacy=0x7ff4f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 >> v1:172.21.15.46:6789/0 conn(0x7ff4f013f710 legacy=0x7ff4f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 1 monclient: found mon.noname-b 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f00b44c0 con 0x7ff4f00a98f0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4f5f1c640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.159+0000 7ff4f5f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.160+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e0001690 con 0x7ff4f00a98f0 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.160+0000 7ff4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.160+0000 7ff4ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:50.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.160+0000 7ff4ee7fc640 10 monclient: dump: 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.160+0000 7ff4f5f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:50.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4e0001960 con 0x7ff4f00a98f0 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 1 -- 172.21.15.46:0/3944971589 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e0001cf0 con 0x7ff4f00a98f0 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4ee7fc640 10 monclient: dump: 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:50.168 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4f5f1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:50.169 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.166+0000 7ff4f5f1c640 4 set_mon_vals no callback set 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 10 monclient: shutdown 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/3944971589 >> v1:172.21.15.46:6790/0 conn(0x7ff4f00a98f0 legacy=0x7ff4f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/3944971589 shutdown_connections 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/3944971589 >> 172.21.15.46:0/3944971589 conn(0x7ff4f00b7ea0 msgr2=0x7ff4f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/3944971589 shutdown_connections 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/3944971589 wait complete. 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.167+0000 7ff4f5f1c640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 10 monclient: monmap: 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:50.170 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 librados: starting msgr at 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 librados: starting objecter 2024-03-25T18:16:50.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 Processor -- start 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 -- start start 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 librados: setting wanted keys 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 1 librados: calling monclient init 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.168+0000 7ff4f5f1c640 10 monclient: init 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f00a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f01430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:50.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f013bc70 con 0x7ff4f00a98f0 2024-03-25T18:16:50.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f013b900 con 0x7ff4f01430c0 2024-03-25T18:16:50.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00b6e30 con 0x7ff4f013f710 2024-03-25T18:16:50.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.169+0000 7ff4f5f1c640 10 monclient(hunting): authenticate will time out at 7719.432337s 2024-03-25T18:16:50.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.172+0000 7ff4effff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff4f01430c0 0x7ff4f01e6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:49152/0 (socket says 172.21.15.46:49152) 2024-03-25T18:16:50.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.172+0000 7ff4effff640 1 -- 172.21.15.46:0/1451456831 learned_addr learned my addr 172.21.15.46:0/1451456831 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:50.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.176+0000 7ff4d7fff640 10 client.?.objecter ms_handle_connect 0x7ff4f01430c0 2024-03-25T18:16:50.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.176+0000 7ff4d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259063948 0 0) 0x7ff4f013b900 con 0x7ff4f01430c0 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:16:50.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.182+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc004eb0 con 0x7ff4f01430c0 2024-03-25T18:16:50.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.189+0000 7ff4d7fff640 10 client.?.objecter ms_handle_connect 0x7ff4f013f710 2024-03-25T18:16:50.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.189+0000 7ff4d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.191+0000 7ff4d7fff640 10 client.?.objecter ms_handle_connect 0x7ff4f00a98f0 2024-03-25T18:16:50.193 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.191+0000 7ff4d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903407209 0 0) 0x7ff4dc004eb0 con 0x7ff4f01430c0 2024-03-25T18:16:50.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f013b900 con 0x7ff4f01430c0 2024-03-25T18:16:50.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e80042e0 con 0x7ff4f01430c0 2024-03-25T18:16:50.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): dump: 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815925182 0 0) 0x7ff4f00b6e30 con 0x7ff4f013f710 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:16:50.205 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.201+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc004eb0 con 0x7ff4f013f710 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108637235 0 0) 0x7ff4f013b900 con 0x7ff4f01430c0 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 >> v1:172.21.15.46:6790/0 conn(0x7ff4f013f710 legacy=0x7ff4f01ded80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 >> v1:172.21.15.46:6789/0 conn(0x7ff4f00a98f0 legacy=0x7ff4f013b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 1 monclient: found mon.b 2024-03-25T18:16:50.215 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f014c270 con 0x7ff4f01430c0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 10 monclient: _renew_subs 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f014c4c0 con 0x7ff4f01430c0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 10 monclient: _renew_subs 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f01df670 con 0x7ff4f01430c0 2024-03-25T18:16:50.216 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.213+0000 7ff4f5f1c640 1 librados: init done 2024-03-25T18:16:50.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.214+0000 7ff4f5f1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.214+0000 7ff4f5f1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.214+0000 7ff4f5f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:50.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.214+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f01df9c0 con 0x7ff4f01430c0 2024-03-25T18:16:50.509 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4e8004830 con 0x7ff4f01430c0 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7ff4e80013f0 con 0x7ff4f01430c0 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4d7fff640 10 monclient: dump: 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.510 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.507+0000 7ff4edffb640 4 set_mon_vals no callback set 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.508+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7ff4e8001610 con 0x7ff4f01430c0 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.509+0000 7ff4ef7fe640 10 monclient: get_auth_request con 0x7ff4dc06dd90 auth_method 0 2024-03-25T18:16:50.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.509+0000 7ff4d7fff640 10 client.4582.objecter ms_handle_connect 0x7ff4dc06dd90 2024-03-25T18:16:50.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.514+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7ff4e8054a00 con 0x7ff4f01430c0 2024-03-25T18:16:50.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.514+0000 7ff4d7fff640 10 client.4582.objecter ms_dispatch 0x7ff4f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:50.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.514+0000 7ff4d7fff640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:50.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.514+0000 7ff4d7fff640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:50.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.514+0000 7ff4d7fff640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:16:50.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.515+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4e8050340 con 0x7ff4f01430c0 2024-03-25T18:16:50.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.515+0000 7ff4d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.515+0000 7ff4d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:50.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.776+0000 7ff4f5f1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.776+0000 7ff4f5f1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.776+0000 7ff4f5f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:50.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.776+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f00b6fc0 con 0x7ff4f01430c0 2024-03-25T18:16:50.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.806+0000 7ff4d7fff640 1 -- 172.21.15.46:0/1451456831 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7ff4e8052440 con 0x7ff4f01430c0 2024-03-25T18:16:50.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.806+0000 7ff4d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.806+0000 7ff4d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:50.808 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:50.808 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 10 librados: watch_flush enter 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 10 librados: watch_flush exit 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 >> v1:172.21.15.164:6800/1612084612 conn(0x7ff4dc06dd90 legacy=0x7ff4dc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 10 monclient: shutdown 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.808+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 >> v1:172.21.15.164:6789/0 conn(0x7ff4f01430c0 legacy=0x7ff4f01e6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.809+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 shutdown_connections 2024-03-25T18:16:50.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.809+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 >> 172.21.15.46:0/1451456831 conn(0x7ff4f00b7ea0 msgr2=0x7ff4f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:50.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.809+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 shutdown_connections 2024-03-25T18:16:50.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.809+0000 7ff4f5f1c640 1 -- 172.21.15.46:0/1451456831 wait complete. 2024-03-25T18:16:50.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:50.809+0000 7ff4f5f1c640 1 librados: shutdown 2024-03-25T18:16:50.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:57.631 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.769+0000 7f7075684640 10 monclient: get_monmap_and_config 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.769+0000 7f7075684640 10 monclient: build_initial_monmap 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.769+0000 7f7075684640 10 monclient: monmap: 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:16:57.770193+0000 2024-03-25T18:16:57.770 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:16:57.770193+0000 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:16:57.771 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.770+0000 7f7075684640 1 Processor -- start 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.770+0000 7f7075684640 1 -- start start 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.770+0000 7f7075684640 10 monclient: init 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient(hunting): picked mon.noname-a con 0x7f70700a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient(hunting): picked mon.noname-b con 0x7f707013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient(hunting): picked mon.noname-c con 0x7f70701430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:57.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7070142520 con 0x7f70700a98f0 2024-03-25T18:16:57.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700b28a0 con 0x7f70701430c0 2024-03-25T18:16:57.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700b2a30 con 0x7f707013f710 2024-03-25T18:16:57.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:57.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.771+0000 7f7075684640 10 monclient(hunting): authenticate will time out at 7727.033016s 2024-03-25T18:16:57.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.777+0000 7f706effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f70700a98f0 0x7f70700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53604/0 (socket says 172.21.15.46:53604) 2024-03-25T18:16:57.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.777+0000 7f706effd640 1 -- 172.21.15.46:0/4035771451 learned_addr learned my addr 172.21.15.46:0/4035771451 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:57.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721402880 0 0) 0x7f7070142520 con 0x7f70700a98f0 2024-03-25T18:16:57.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.793 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:16:57.794 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.792+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7058003520 con 0x7f70700a98f0 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014405593 0 0) 0x7f70700b28a0 con 0x7f70701430c0 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:16:57.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.801+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7070142520 con 0x7f70701430c0 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016131641 0 0) 0x7f7058003520 con 0x7f70700a98f0 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 >> v1:172.21.15.46:6790/0 conn(0x7f707013f710 legacy=0x7f70701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 >> v1:172.21.15.164:6789/0 conn(0x7f70701430c0 legacy=0x7f707014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 monclient: found mon.noname-a 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:16:57.806 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700b44c0 con 0x7f70700a98f0 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7060001690 con 0x7f70700a98f0 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f7075684640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f7075684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f706dffb640 10 monclient: dump: 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.807 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:57.808 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:57.808 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:57.808 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:57.808 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:57.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.805+0000 7f7075684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:57.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.822+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70600017f0 con 0x7f70700a98f0 2024-03-25T18:16:57.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.822+0000 7f706dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:57.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f706dffb640 1 -- 172.21.15.46:0/4035771451 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7060001c00 con 0x7f70700a98f0 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f7075684640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f7075684640 4 set_mon_vals no callback set 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f706dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f706dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f706dffb640 10 monclient: dump: 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:57.824 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:57.825 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f7075684640 10 monclient: shutdown 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f7075684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.823+0000 7f7075684640 1 -- 172.21.15.46:0/4035771451 >> v1:172.21.15.46:6789/0 conn(0x7f70700a98f0 legacy=0x7f70700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 1 -- 172.21.15.46:0/4035771451 shutdown_connections 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 1 -- 172.21.15.46:0/4035771451 >> 172.21.15.46:0/4035771451 conn(0x7f70700b7ea0 msgr2=0x7f70700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 1 -- 172.21.15.46:0/4035771451 shutdown_connections 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 1 -- 172.21.15.46:0/4035771451 wait complete. 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 10 monclient: build_initial_monmap 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.824+0000 7f7075684640 10 monclient: monmap: 2024-03-25T18:16:57.827 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 librados: starting msgr at 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 librados: starting objecter 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 Processor -- start 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 -- start start 2024-03-25T18:16:57.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 librados: setting wanted keys 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 librados: calling monclient init 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient: init 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient(hunting): picked mon.noname-c con 0x7f70700a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient(hunting): picked mon.noname-b con 0x7f707013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient(hunting): picked mon.noname-a con 0x7f70701430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700b7ce0 con 0x7f70701430c0 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70701319b0 con 0x7f707013f710 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700b7800 con 0x7f70700a98f0 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:57.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.825+0000 7f7075684640 10 monclient(hunting): authenticate will time out at 7727.087528s 2024-03-25T18:16:57.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.833+0000 7f706e7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f707013f710 0x7f7070130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:46570/0 (socket says 172.21.15.46:46570) 2024-03-25T18:16:57.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.833+0000 7f706e7fc640 1 -- 172.21.15.46:0/1103879107 learned_addr learned my addr 172.21.15.46:0/1103879107 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:16:57.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.835+0000 7f70577fe640 10 client.?.objecter ms_handle_connect 0x7f707013f710 2024-03-25T18:16:57.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.835+0000 7f70577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392426417 0 0) 0x7f70701319b0 con 0x7f707013f710 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:16:57.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.842+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f705c004eb0 con 0x7f707013f710 2024-03-25T18:16:57.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.850+0000 7f70577fe640 10 client.?.objecter ms_handle_connect 0x7f70700a98f0 2024-03-25T18:16:57.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.850+0000 7f70577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:57.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.852+0000 7f70577fe640 10 client.?.objecter ms_handle_connect 0x7f70701430c0 2024-03-25T18:16:57.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.852+0000 7f70577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970759187 0 0) 0x7f705c004eb0 con 0x7f707013f710 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70701319b0 con 0x7f707013f710 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7064001a30 con 0x7f707013f710 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 10 monclient(hunting): dump: 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.859 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:57.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.858+0000 7f70577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239057510 0 0) 0x7f70700b7800 con 0x7f70700a98f0 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:16:57.868 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.867+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f705c004eb0 con 0x7f70700a98f0 2024-03-25T18:16:57.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701527384 0 0) 0x7f70700b7ce0 con 0x7f70701430c0 2024-03-25T18:16:57.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.872 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.872 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.872 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:16:57.872 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.870+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70700b7800 con 0x7f70701430c0 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3958353900 0 0) 0x7f70701319b0 con 0x7f707013f710 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 >> v1:172.21.15.46:6790/0 conn(0x7f70700a98f0 legacy=0x7f70701dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 >> v1:172.21.15.46:6789/0 conn(0x7f70701430c0 legacy=0x7f70701312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 1 monclient: found mon.b 2024-03-25T18:16:57.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070136f50 con 0x7f707013f710 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f70577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 10 monclient: _renew_subs 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70701350f0 con 0x7f707013f710 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 10 monclient: _renew_subs 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070135c60 con 0x7f707013f710 2024-03-25T18:16:57.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.883+0000 7f7075684640 1 librados: init done 2024-03-25T18:16:57.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.884+0000 7f7075684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:57.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.884+0000 7f7075684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:57.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.884+0000 7f7075684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:57.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:57.884+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70700627b0 con 0x7f707013f710 2024-03-25T18:16:58.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7064001cb0 con 0x7f707013f710 2024-03-25T18:16:58.190 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7064003a60 con 0x7f707013f710 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f70577fe640 10 monclient: dump: 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:16:58.191 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:16:58.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.189+0000 7f706d7fa640 4 set_mon_vals no callback set 2024-03-25T18:16:58.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.190+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7064003c40 con 0x7f707013f710 2024-03-25T18:16:58.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.191+0000 7f706effd640 10 monclient: get_auth_request con 0x7f705c06de80 auth_method 0 2024-03-25T18:16:58.192 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.191+0000 7f70577fe640 10 client.4588.objecter ms_handle_connect 0x7f705c06de80 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f7064054900 con 0x7f707013f710 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 10 client.4588.objecter ms_dispatch 0x7f7070000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7064050410 con 0x7f707013f710 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.195+0000 7f70577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.449+0000 7f7075684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.449+0000 7f7075684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.449+0000 7f7075684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:16:58.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.449+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f707013e270 con 0x7f707013f710 2024-03-25T18:16:58.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.475+0000 7f70577fe640 1 -- 172.21.15.46:0/1103879107 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f7064052340 con 0x7f707013f710 2024-03-25T18:16:58.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.475+0000 7f70577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.475+0000 7f70577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:58.476 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:16:58.476 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 10 librados: watch_flush enter 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 10 librados: watch_flush exit 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 10 client.4588.objecter successfully canceled tick 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 >> v1:172.21.15.164:6800/1612084612 conn(0x7f705c06de80 legacy=0x7f705c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 10 monclient: shutdown 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 >> v1:172.21.15.164:6789/0 conn(0x7f707013f710 legacy=0x7f7070130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.477+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 shutdown_connections 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.478+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 >> 172.21.15.46:0/1103879107 conn(0x7f70700b7ea0 msgr2=0x7f7070074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:58.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.478+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 shutdown_connections 2024-03-25T18:16:58.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.478+0000 7f7075684640 1 -- 172.21.15.46:0/1103879107 wait complete. 2024-03-25T18:16:58.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:16:58.478+0000 7f7075684640 1 librados: shutdown 2024-03-25T18:16:58.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:05.297 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:05.392 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:41.310 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.410+0000 7fd1e817c640 10 monclient: get_monmap_and_config 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.410+0000 7fd1e817c640 10 monclient: build_initial_monmap 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.410+0000 7fd1e817c640 10 monclient: monmap: 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:05.411342+0000 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:05.411342+0000 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.411 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:05.412 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:05.412 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:05.412 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.411+0000 7fd1e817c640 1 Processor -- start 2024-03-25T18:17:05.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.411+0000 7fd1e817c640 1 -- start start 2024-03-25T18:17:05.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.411+0000 7fd1e817c640 10 monclient: init 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.411+0000 7fd1e817c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.411+0000 7fd1e817c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e014be90 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00af940 con 0x7fd1e00b4d60 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00b0a70 con 0x7fd1e014be90 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00b0d50 con 0x7fd1e00b1730 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e817c640 10 monclient(hunting): authenticate will time out at 7734.673650s 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e5f18640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd1e00b4d60 0x7fd1e013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52896/0 (socket says 172.21.15.46:52896) 2024-03-25T18:17:05.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.412+0000 7fd1e5f18640 1 -- 172.21.15.46:0/1326924796 learned_addr learned my addr 172.21.15.46:0/1326924796 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:05.442 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:41.360 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602334477 0 0) 0x7fd1e00b0d50 con 0x7fd1e00b1730 2024-03-25T18:17:05.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 10 monclient(hunting): _negotiate 2024-03-25T18:17:05.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:05.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:05.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:17:05.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.474+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc003520 con 0x7fd1e00b1730 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634640022 0 0) 0x7fd1cc003520 con 0x7fd1e00b1730 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 >> v1:172.21.15.164:6789/0 conn(0x7fd1e014be90 legacy=0x7fd1e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 >> v1:172.21.15.46:6789/0 conn(0x7fd1e00b4d60 legacy=0x7fd1e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: _finish_hunting 0 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 monclient: found mon.noname-b 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e00b1030 con 0x7fd1e00b1730 2024-03-25T18:17:05.488 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: _finish_auth 0 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: _check_auth_tickets 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd1d0001690 con 0x7fd1e00b1730 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e817c640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e817c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e4f16640 10 monclient: dump: 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:05.489 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:05.490 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:05.490 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.487+0000 7fd1e817c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:05.490 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:08:41.405 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.500+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1d00017f0 con 0x7fd1e00b1730 2024-03-25T18:17:05.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e4f16640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:05.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e4f16640 1 -- 172.21.15.46:0/1326924796 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd1d0001c00 con 0x7fd1e00b1730 2024-03-25T18:17:05.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:05.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 4 set_mon_vals no callback set 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e4f16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e4f16640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e4f16640 10 monclient: dump: 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 10 monclient: shutdown 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 1 -- 172.21.15.46:0/1326924796 >> v1:172.21.15.46:6790/0 conn(0x7fd1e00b1730 legacy=0x7fd1e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 1 -- 172.21.15.46:0/1326924796 shutdown_connections 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 1 -- 172.21.15.46:0/1326924796 >> 172.21.15.46:0/1326924796 conn(0x7fd1e00617a0 msgr2=0x7fd1e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 1 -- 172.21.15.46:0/1326924796 shutdown_connections 2024-03-25T18:17:05.504 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.501+0000 7fd1e817c640 1 -- 172.21.15.46:0/1326924796 wait complete. 2024-03-25T18:17:05.505 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:41.373 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.502+0000 7fd1e817c640 10 monclient: build_initial_monmap 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.502+0000 7fd1e817c640 10 monclient: monmap: 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:05.506 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 librados: starting msgr at 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 librados: starting objecter 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 Processor -- start 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 -- start start 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 librados: setting wanted keys 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 1 librados: calling monclient init 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.503+0000 7fd1e817c640 10 monclient: init 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e014be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e0138250 con 0x7fd1e00b4d60 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e01383e0 con 0x7fd1e00b1730 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e01387a0 con 0x7fd1e014be90 2024-03-25T18:17:05.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:05.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.504+0000 7fd1e817c640 10 monclient(hunting): authenticate will time out at 7734.765763s 2024-03-25T18:17:05.509 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:41.380 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.515+0000 7fd1e5f18640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd1e00b1730 0x7fd1e0137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:33030/0 (socket says 172.21.15.46:33030) 2024-03-25T18:17:05.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.515+0000 7fd1e5f18640 1 -- 172.21.15.46:0/3218966528 learned_addr learned my addr 172.21.15.46:0/3218966528 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:05.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.520+0000 7fd1d67fc640 10 client.?.objecter ms_handle_connect 0x7fd1e00b1730 2024-03-25T18:17:05.520 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.520+0000 7fd1d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:05.521 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:41.439 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.526 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:41.444 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28153260 0 0) 0x7fd1e01383e0 con 0x7fd1e00b1730 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:17:05.531 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.530+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4004eb0 con 0x7fd1e00b1730 2024-03-25T18:17:05.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.534+0000 7fd1d67fc640 10 client.?.objecter ms_handle_connect 0x7fd1e00b4d60 2024-03-25T18:17:05.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.534+0000 7fd1d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:05.546 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:08:41.417 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3105539616 0 0) 0x7fd1c4004eb0 con 0x7fd1e00b1730 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e01383e0 con 0x7fd1e00b1730 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd1dc0018b0 con 0x7fd1e00b1730 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 10 monclient(hunting): dump: 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.554 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.553+0000 7fd1d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:05.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550293379 0 0) 0x7fd1e0138250 con 0x7fd1e00b4d60 2024-03-25T18:17:05.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:05.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:05.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:05.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:17:05.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.554+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4004eb0 con 0x7fd1e00b4d60 2024-03-25T18:17:05.576 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:08:41.493 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:05.585 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.585+0000 7fd1d67fc640 10 client.?.objecter ms_handle_connect 0x7fd1e014be90 2024-03-25T18:17:05.586 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.585+0000 7fd1d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:05.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722367920 0 0) 0x7fd1c4004eb0 con 0x7fd1e00b4d60 2024-03-25T18:17:05.600 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e0138250 con 0x7fd1e00b4d60 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd1d0001bc0 con 0x7fd1e00b4d60 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.600+0000 7fd1d67fc640 10 monclient(hunting): dump: 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:05.601 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1836018079 0 0) 0x7fd1e01383e0 con 0x7fd1e00b1730 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 >> v1:172.21.15.46:6790/0 conn(0x7fd1e014be90 legacy=0x7fd1e0130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 >> v1:172.21.15.46:6789/0 conn(0x7fd1e00b4d60 legacy=0x7fd1e01db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 1 monclient: found mon.b 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0138930 con 0x7fd1e00b1730 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:05.622 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 10 monclient: _renew_subs 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e0143b60 con 0x7fd1e00b1730 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 10 monclient: _renew_subs 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e01dbb50 con 0x7fd1e00b1730 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.620+0000 7fd1e817c640 1 librados: init done 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.621+0000 7fd1e817c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.621+0000 7fd1e817c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.621+0000 7fd1e817c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:05.623 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.621+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e01db0b0 con 0x7fd1e00b1730 2024-03-25T18:17:05.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc0047e0 con 0x7fd1e00b1730 2024-03-25T18:17:05.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:05.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd1dc004bf0 con 0x7fd1e00b1730 2024-03-25T18:17:05.899 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d67fc640 10 monclient: dump: 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:05.900 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.898+0000 7fd1d7fff640 4 set_mon_vals no callback set 2024-03-25T18:17:05.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.899+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fd1dc004e10 con 0x7fd1e00b1730 2024-03-25T18:17:05.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.900+0000 7fd1e5717640 10 monclient: get_auth_request con 0x7fd1c406deb0 auth_method 0 2024-03-25T18:17:05.901 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.900+0000 7fd1d67fc640 10 client.4594.objecter ms_handle_connect 0x7fd1c406deb0 2024-03-25T18:17:05.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fd1dc0548a0 con 0x7fd1e00b1730 2024-03-25T18:17:05.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 10 client.4594.objecter ms_dispatch 0x7fd1e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:05.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:05.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:05.906 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:17:05.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1dc0503b0 con 0x7fd1e00b1730 2024-03-25T18:17:05.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:05.907 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:05.905+0000 7fd1d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:06.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.135+0000 7fd1e817c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.135+0000 7fd1e817c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.135+0000 7fd1e817c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:06.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.135+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1e00627b0 con 0x7fd1e00b1730 2024-03-25T18:17:06.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.161+0000 7fd1d67fc640 1 -- 172.21.15.46:0/3218966528 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fd1dc0522e0 con 0x7fd1e00b1730 2024-03-25T18:17:06.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.161+0000 7fd1d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.162 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.161+0000 7fd1d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:06.162 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:06.162 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.163+0000 7fd1e817c640 10 librados: watch_flush enter 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.163+0000 7fd1e817c640 10 librados: watch_flush exit 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.163+0000 7fd1e817c640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.163+0000 7fd1e817c640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 >> v1:172.21.15.164:6800/1612084612 conn(0x7fd1c406deb0 legacy=0x7fd1c4070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 10 monclient: shutdown 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 >> v1:172.21.15.164:6789/0 conn(0x7fd1e00b1730 legacy=0x7fd1e0137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 shutdown_connections 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 >> 172.21.15.46:0/3218966528 conn(0x7fd1e00617a0 msgr2=0x7fd1e00b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 shutdown_connections 2024-03-25T18:17:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 -- 172.21.15.46:0/3218966528 wait complete. 2024-03-25T18:17:06.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:06.164+0000 7fd1e817c640 1 librados: shutdown 2024-03-25T18:17:06.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:12.984 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.105+0000 7fd3b18ff640 10 monclient: get_monmap_and_config 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.105+0000 7fd3b18ff640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.105+0000 7fd3b18ff640 10 monclient: monmap: 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:13.105813+0000 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:13.105813+0000 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:13.106 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:13.107 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:13.107 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:13.107 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 1 Processor -- start 2024-03-25T18:17:13.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 1 -- start start 2024-03-25T18:17:13.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient: init 2024-03-25T18:17:13.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac0b4d60 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac0b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac14be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0af940 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0b0a70 con 0x7fd3ac0b1730 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0b0d50 con 0x7fd3ac14be90 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.106+0000 7fd3b18ff640 10 monclient(hunting): authenticate will time out at 7742.368271s 2024-03-25T18:17:13.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.112+0000 7fd3aaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3ac0b4d60 0x7fd3ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57542/0 (socket says 172.21.15.46:57542) 2024-03-25T18:17:13.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.112+0000 7fd3aaffd640 1 -- 172.21.15.46:0/3187566 learned_addr learned my addr 172.21.15.46:0/3187566 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430540348 0 0) 0x7fd3ac0b0d50 con 0x7fd3ac14be90 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:17:13.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.126+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd398003520 con 0x7fd3ac14be90 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27182225 0 0) 0x7fd3ac0b0a70 con 0x7fd3ac0b1730 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:17:13.131 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.130+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac0b0d50 con 0x7fd3ac0b1730 2024-03-25T18:17:13.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708517118 0 0) 0x7fd3ac0af940 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:17:13.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.134+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac0b0a70 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512256408 0 0) 0x7fd398003520 con 0x7fd3ac14be90 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 >> v1:172.21.15.164:6789/0 conn(0x7fd3ac0b1730 legacy=0x7fd3ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 >> v1:172.21.15.46:6789/0 conn(0x7fd3ac0b4d60 legacy=0x7fd3ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0b1030 con 0x7fd3ac14be90 2024-03-25T18:17:13.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd3a4001a10 con 0x7fd3ac14be90 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3aa7fc640 10 monclient: dump: 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:13.143 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:13.144 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3b18ff640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:17:13.144 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.141+0000 7fd3b18ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:13.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3a4001b70 con 0x7fd3ac14be90 2024-03-25T18:17:13.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:13.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 1 -- 172.21.15.46:0/3187566 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd3a4001f80 con 0x7fd3ac14be90 2024-03-25T18:17:13.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3b18ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:13.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3b18ff640 4 set_mon_vals no callback set 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.159+0000 7fd3aa7fc640 10 monclient: dump: 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:13.161 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 10 monclient: shutdown 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 1 -- 172.21.15.46:0/3187566 >> v1:172.21.15.46:6790/0 conn(0x7fd3ac14be90 legacy=0x7fd3ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 1 -- 172.21.15.46:0/3187566 shutdown_connections 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 1 -- 172.21.15.46:0/3187566 >> 172.21.15.46:0/3187566 conn(0x7fd3ac0617a0 msgr2=0x7fd3ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 1 -- 172.21.15.46:0/3187566 shutdown_connections 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.160+0000 7fd3b18ff640 1 -- 172.21.15.46:0/3187566 wait complete. 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.164 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 10 monclient: monmap: 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 librados: starting msgr at 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 librados: starting objecter 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 Processor -- start 2024-03-25T18:17:13.165 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 -- start start 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 librados: setting wanted keys 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 1 librados: calling monclient init 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.161+0000 7fd3b18ff640 10 monclient: init 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac0b1730 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac0b4d60 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac14be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac13aa60 con 0x7fd3ac14be90 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac143210 con 0x7fd3ac0b1730 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac1433a0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.162+0000 7fd3b18ff640 10 monclient(hunting): authenticate will time out at 7742.424162s 2024-03-25T18:17:13.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.175+0000 7fd3a3fff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fd3ac0b4d60 0x7fd3ac1d7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:54634/0 (socket says 172.21.15.46:54634) 2024-03-25T18:17:13.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.175+0000 7fd3a3fff640 1 -- 172.21.15.46:0/2841630449 learned_addr learned my addr 172.21.15.46:0/2841630449 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:13.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.180+0000 7fd3a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd3ac0b1730 2024-03-25T18:17:13.181 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.180+0000 7fd3a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.181+0000 7fd3a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd3ac0b4d60 2024-03-25T18:17:13.182 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.182+0000 7fd3a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866454644 0 0) 0x7fd3ac143210 con 0x7fd3ac0b1730 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:17:13.188 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd38c004eb0 con 0x7fd3ac0b1730 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd3ac14be90 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.187+0000 7fd3a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126627595 0 0) 0x7fd3ac1433a0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:17:13.189 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.188+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac143210 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.199+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 568604195 0 0) 0x7fd3ac143210 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3ac1433a0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd3940019b0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:13.201 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:13.202 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.200+0000 7fd3a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:13.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668023201 0 0) 0x7fd3ac13aa60 con 0x7fd3ac14be90 2024-03-25T18:17:13.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.206 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:17:13.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.205+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac143210 con 0x7fd3ac14be90 2024-03-25T18:17:13.209 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 100362570 0 0) 0x7fd3ac1433a0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 >> v1:172.21.15.164:6789/0 conn(0x7fd3ac0b1730 legacy=0x7fd3ac139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 >> v1:172.21.15.46:6789/0 conn(0x7fd3ac14be90 legacy=0x7fd3ac1e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 1 monclient: found mon.c 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac139850 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.210 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 5 monclient: authenticate success, global_id 4616 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 10 monclient: _renew_subs 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac13ae80 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 10 monclient: _renew_subs 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac1d8400 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.211 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.208+0000 7fd3b18ff640 1 librados: init done 2024-03-25T18:17:13.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.210+0000 7fd3b18ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.212 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.210+0000 7fd3b18ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.210+0000 7fd3b18ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.213 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.210+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac1d77b0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd394003610 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd3940039a0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:13.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a8ff9640 10 monclient: dump: 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:13.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.535+0000 7fd3a9ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:13.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.536+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fd394003bc0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.537+0000 7fd3aaffd640 10 monclient: get_auth_request con 0x7fd38c06de10 auth_method 0 2024-03-25T18:17:13.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.537+0000 7fd3a8ff9640 10 client.4616.objecter ms_handle_connect 0x7fd38c06de10 2024-03-25T18:17:13.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fd394054800 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 10 client.4616.objecter ms_dispatch 0x7fd3ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:13.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:13.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:13.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T18:17:13.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd394050310 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.544+0000 7fd3a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:13.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.794+0000 7fd3b18ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.794+0000 7fd3b18ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.794+0000 7fd3b18ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:13.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.794+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3ac139ac0 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.824+0000 7fd3a8ff9640 1 -- 172.21.15.46:0/2841630449 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fd394052240 con 0x7fd3ac0b4d60 2024-03-25T18:17:13.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.824+0000 7fd3a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.824+0000 7fd3a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:13.825 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:13.825 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:13.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 10 librados: watch_flush enter 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 10 librados: watch_flush exit 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 10 client.4616.objecter successfully canceled tick 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 >> v1:172.21.15.164:6800/1612084612 conn(0x7fd38c06de10 legacy=0x7fd38c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 10 monclient: shutdown 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 >> v1:172.21.15.46:6790/0 conn(0x7fd3ac0b4d60 legacy=0x7fd3ac1d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 shutdown_connections 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.826+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 >> 172.21.15.46:0/2841630449 conn(0x7fd3ac0617a0 msgr2=0x7fd3ac0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.827+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 shutdown_connections 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.827+0000 7fd3b18ff640 1 -- 172.21.15.46:0/2841630449 wait complete. 2024-03-25T18:17:13.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:13.827+0000 7fd3b18ff640 1 librados: shutdown 2024-03-25T18:17:13.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:20.645 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:20.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 10 monclient: get_monmap_and_config 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 10 monclient: build_initial_monmap 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 10 monclient: monmap: 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:20.760841+0000 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:20.760841+0000 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:20.761 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:20.762 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:20.762 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 1 Processor -- start 2024-03-25T18:17:20.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 1 -- start start 2024-03-25T18:17:20.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.760+0000 7f96f4788640 10 monclient: init 2024-03-25T18:17:20.763 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.761+0000 7f96f4788640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.761+0000 7f96f4788640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.761+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec0705b0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.761+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec1402d0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec149ae0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0718c0 con 0x7f96ec149ae0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0714c0 con 0x7f96ec1402d0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec072b50 con 0x7f96ec0705b0 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:20.764 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.762+0000 7f96f4788640 10 monclient(hunting): authenticate will time out at 7750.023660s 2024-03-25T18:17:20.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.775+0000 7f96f1d23640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f96ec1402d0 0x7f96ec140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:49068/0 (socket says 172.21.15.46:49068) 2024-03-25T18:17:20.776 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.775+0000 7f96f1d23640 1 -- 172.21.15.46:0/245921104 learned_addr learned my addr 172.21.15.46:0/245921104 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:20.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146248168 0 0) 0x7f96ec0714c0 con 0x7f96ec1402d0 2024-03-25T18:17:20.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:17:20.784 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.782+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d8003520 con 0x7f96ec1402d0 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971144574 0 0) 0x7f96ec072b50 con 0x7f96ec0705b0 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec0714c0 con 0x7f96ec0705b0 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305923503 0 0) 0x7f96ec0718c0 con 0x7f96ec149ae0 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.798 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.798 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:17:20.798 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.796+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec072b50 con 0x7f96ec149ae0 2024-03-25T18:17:20.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275234262 0 0) 0x7f96d8003520 con 0x7f96ec1402d0 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 >> v1:172.21.15.46:6790/0 conn(0x7f96ec0705b0 legacy=0x7f96ec13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 >> v1:172.21.15.46:6789/0 conn(0x7f96ec149ae0 legacy=0x7f96ec14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: _finish_hunting 0 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 monclient: found mon.noname-c 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec13dd10 con 0x7f96ec1402d0 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: _finish_auth 0 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: _check_auth_tickets 2024-03-25T18:17:20.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96e8001690 con 0x7f96ec1402d0 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f1522640 10 monclient: dump: 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:20.803 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:20.804 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f4788640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:17:20.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.800+0000 7f96f4788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96e80017f0 con 0x7f96ec1402d0 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 1 -- 172.21.15.46:0/245921104 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96e8001c00 con 0x7f96ec1402d0 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f4788640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f4788640 4 set_mon_vals no callback set 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.818+0000 7f96f1522640 10 monclient: dump: 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.819 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:20.820 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 10 monclient: shutdown 2024-03-25T18:17:20.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 1 -- 172.21.15.46:0/245921104 >> v1:172.21.15.164:6789/0 conn(0x7f96ec1402d0 legacy=0x7f96ec140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 1 -- 172.21.15.46:0/245921104 shutdown_connections 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 1 -- 172.21.15.46:0/245921104 >> 172.21.15.46:0/245921104 conn(0x7f96ec0b5d10 msgr2=0x7f96ec0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 1 -- 172.21.15.46:0/245921104 shutdown_connections 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.819+0000 7f96f4788640 1 -- 172.21.15.46:0/245921104 wait complete. 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 10 monclient: build_initial_monmap 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 10 monclient: monmap: 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:20.822 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 librados: starting msgr at 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 librados: starting objecter 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 Processor -- start 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 -- start start 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 librados: setting wanted keys 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 1 librados: calling monclient init 2024-03-25T18:17:20.823 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.820+0000 7f96f4788640 10 monclient: init 2024-03-25T18:17:20.824 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec0705b0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec1402d0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec149ae0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec13edf0 con 0x7f96ec0705b0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec06bc90 con 0x7f96ec149ae0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec06bf70 con 0x7f96ec1402d0 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:20.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.821+0000 7f96f4788640 10 monclient(hunting): authenticate will time out at 7750.083485s 2024-03-25T18:17:20.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.834+0000 7f96f2524640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96ec0705b0 0x7f96ec132f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37654/0 (socket says 172.21.15.46:37654) 2024-03-25T18:17:20.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.834+0000 7f96f2524640 1 -- 172.21.15.46:0/2720588955 learned_addr learned my addr 172.21.15.46:0/2720588955 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:20.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.838+0000 7f96e2ffd640 10 client.?.objecter ms_handle_connect 0x7f96ec149ae0 2024-03-25T18:17:20.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.838+0000 7f96e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.839+0000 7f96e2ffd640 10 client.?.objecter ms_handle_connect 0x7f96ec0705b0 2024-03-25T18:17:20.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.839+0000 7f96e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.844 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.844+0000 7f96e2ffd640 10 client.?.objecter ms_handle_connect 0x7f96ec1402d0 2024-03-25T18:17:20.845 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.844+0000 7f96e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964416436 0 0) 0x7f96ec06bf70 con 0x7f96ec1402d0 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d0004eb0 con 0x7f96ec1402d0 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493471951 0 0) 0x7f96ec13edf0 con 0x7f96ec0705b0 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.849+0000 7f96e2ffd640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:17:20.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec06bf70 con 0x7f96ec0705b0 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295224640 0 0) 0x7f96ec06bc90 con 0x7f96ec149ae0 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:17:20.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.850+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec13edf0 con 0x7f96ec149ae0 2024-03-25T18:17:20.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2409717583 0 0) 0x7f96ec06bf70 con 0x7f96ec0705b0 2024-03-25T18:17:20.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec06bc90 con 0x7f96ec0705b0 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96dc001930 con 0x7f96ec0705b0 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:20.866 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:20.867 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.864+0000 7f96e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824991455 0 0) 0x7f96ec13edf0 con 0x7f96ec149ae0 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec06bf70 con 0x7f96ec149ae0 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96e4001bd0 con 0x7f96ec149ae0 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.868+0000 7f96e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:20.869 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:20.870 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293001776 0 0) 0x7f96d0004eb0 con 0x7f96ec1402d0 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec13edf0 con 0x7f96ec1402d0 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96e8001b40 con 0x7f96ec1402d0 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.869+0000 7f96e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.871 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:20.872 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:20.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.883+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 170924093 0 0) 0x7f96ec06bf70 con 0x7f96ec149ae0 2024-03-25T18:17:20.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 >> v1:172.21.15.46:6790/0 conn(0x7f96ec1402d0 legacy=0x7f96ec13dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 >> v1:172.21.15.46:6789/0 conn(0x7f96ec0705b0 legacy=0x7f96ec132f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:20.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 1 monclient: found mon.b 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec1349f0 con 0x7f96ec149ae0 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 10 monclient: _renew_subs 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec06ca40 con 0x7f96ec149ae0 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 10 monclient: _renew_subs 2024-03-25T18:17:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec06cf60 con 0x7f96ec149ae0 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.884+0000 7f96f4788640 1 librados: init done 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.885+0000 7f96f4788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.885+0000 7f96f4788640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.885+0000 7f96f4788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:20.885+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec0627b0 con 0x7f96ec149ae0 2024-03-25T18:17:21.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.170+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96e4001fe0 con 0x7f96ec149ae0 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f96e4003b30 con 0x7f96ec149ae0 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 10 monclient: dump: 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.172 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96f0d21640 4 set_mon_vals no callback set 2024-03-25T18:17:21.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.171+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f96e4019b00 con 0x7f96ec149ae0 2024-03-25T18:17:21.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.172+0000 7f96f2524640 10 monclient: get_auth_request con 0x7f96d006e160 auth_method 0 2024-03-25T18:17:21.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.172+0000 7f96e2ffd640 10 client.4606.objecter ms_handle_connect 0x7f96d006e160 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f96e40536c0 con 0x7f96ec149ae0 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 10 client.4606.objecter ms_dispatch 0x7f96ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96e404f1d0 con 0x7f96ec149ae0 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.178+0000 7f96e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:21.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.413+0000 7f96f4788640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.413+0000 7f96f4788640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.413+0000 7f96f4788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:17:21.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.413+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ec06cbd0 con 0x7f96ec149ae0 2024-03-25T18:17:21.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.496+0000 7f96e2ffd640 1 -- 172.21.15.46:0/2720588955 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f96e4051100 con 0x7f96ec149ae0 2024-03-25T18:17:21.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.496+0000 7f96e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.496+0000 7f96e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:21.498 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:21.498 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 10 librados: watch_flush enter 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 10 librados: watch_flush exit 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 >> v1:172.21.15.164:6800/1612084612 conn(0x7f96d006e160 legacy=0x7f96d0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 10 monclient: shutdown 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 >> v1:172.21.15.164:6789/0 conn(0x7f96ec149ae0 legacy=0x7f96ec13e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 shutdown_connections 2024-03-25T18:17:21.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.498+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 >> 172.21.15.46:0/2720588955 conn(0x7f96ec0b5d10 msgr2=0x7f96ec13b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.499+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 shutdown_connections 2024-03-25T18:17:21.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.499+0000 7f96f4788640 1 -- 172.21.15.46:0/2720588955 wait complete. 2024-03-25T18:17:21.501 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:21.499+0000 7f96f4788640 1 librados: shutdown 2024-03-25T18:17:21.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:25.489 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:01.406 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.499 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:01.417 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.531 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:01.448 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.563 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:01.434 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.571 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:01.488 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.571 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:01.442 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.580 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:01.498 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.591 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:01.462 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.614 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:01.531 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.319 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:28.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.435+0000 7fd37379a640 10 monclient: get_monmap_and_config 2024-03-25T18:17:28.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.435+0000 7fd37379a640 10 monclient: build_initial_monmap 2024-03-25T18:17:28.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.435+0000 7fd37379a640 10 monclient: monmap: 2024-03-25T18:17:28.448 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:28.448 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:28.435841+0000 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:28.435841+0000 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.435+0000 7fd37379a640 1 Processor -- start 2024-03-25T18:17:28.449 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.436+0000 7fd37379a640 1 -- start start 2024-03-25T18:17:28.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.436+0000 7fd37379a640 10 monclient: init 2024-03-25T18:17:28.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:28.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:28.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-c con 0x7fd36c0a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:28.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c13f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c142520 con 0x7fd36c13f710 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c0b28a0 con 0x7fd36c0a98f0 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c0b2a30 con 0x7fd36c1430c0 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.437+0000 7fd37379a640 10 monclient(hunting): authenticate will time out at 7757.698709s 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.447+0000 7fd370d35640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd36c13f710 0x7fd36c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36140/0 (socket says 172.21.15.46:36140) 2024-03-25T18:17:28.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.447+0000 7fd370d35640 1 -- 172.21.15.46:0/1613021077 learned_addr learned my addr 172.21.15.46:0/1613021077 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:28.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320566453 0 0) 0x7fd36c142520 con 0x7fd36c13f710 2024-03-25T18:17:28.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:17:28.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.459+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358003520 con 0x7fd36c13f710 2024-03-25T18:17:28.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916221614 0 0) 0x7fd36c0b28a0 con 0x7fd36c0a98f0 2024-03-25T18:17:28.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:17:28.468 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.466+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c142520 con 0x7fd36c0a98f0 2024-03-25T18:17:28.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610205643 0 0) 0x7fd358003520 con 0x7fd36c13f710 2024-03-25T18:17:28.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 >> v1:172.21.15.46:6790/0 conn(0x7fd36c1430c0 legacy=0x7fd36c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 >> v1:172.21.15.164:6789/0 conn(0x7fd36c0a98f0 legacy=0x7fd36c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 monclient: found mon.noname-a 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c0b44c0 con 0x7fd36c13f710 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd35c001690 con 0x7fd36c13f710 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd37379a640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:17:28.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd37379a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd363fff640 10 monclient: dump: 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:28.473 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:28.474 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:28.474 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.469+0000 7fd37379a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd35c0017f0 con 0x7fd36c13f710 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 1 -- 172.21.15.46:0/1613021077 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd35c001c00 con 0x7fd36c13f710 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd37379a640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd37379a640 4 set_mon_vals no callback set 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.475+0000 7fd363fff640 10 monclient: dump: 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:28.477 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 10 monclient: shutdown 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 1 -- 172.21.15.46:0/1613021077 >> v1:172.21.15.46:6789/0 conn(0x7fd36c13f710 legacy=0x7fd36c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 1 -- 172.21.15.46:0/1613021077 shutdown_connections 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 1 -- 172.21.15.46:0/1613021077 >> 172.21.15.46:0/1613021077 conn(0x7fd36c0b7ea0 msgr2=0x7fd36c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 1 -- 172.21.15.46:0/1613021077 shutdown_connections 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.476+0000 7fd37379a640 1 -- 172.21.15.46:0/1613021077 wait complete. 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 10 monclient: build_initial_monmap 2024-03-25T18:17:28.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 10 monclient: monmap: 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 librados: starting msgr at 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 librados: starting objecter 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 Processor -- start 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 -- start start 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 librados: setting wanted keys 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 1 librados: calling monclient init 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 10 monclient: init 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.477+0000 7fd37379a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:28.483 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient(hunting): picked mon.noname-c con 0x7fd36c1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c1d7490 con 0x7fd36c0a98f0 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c1d8360 con 0x7fd36c13f710 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c0abb90 con 0x7fd36c1430c0 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:28.484 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.478+0000 7fd37379a640 10 monclient(hunting): authenticate will time out at 7757.739787s 2024-03-25T18:17:28.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.486+0000 7fd370d35640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd36c13f710 0x7fd36c1d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:35010/0 (socket says 172.21.15.46:35010) 2024-03-25T18:17:28.487 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.486+0000 7fd370d35640 1 -- 172.21.15.46:0/1036354528 learned_addr learned my addr 172.21.15.46:0/1036354528 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:28.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.490+0000 7fd361ffb640 10 client.?.objecter ms_handle_connect 0x7fd36c13f710 2024-03-25T18:17:28.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.490+0000 7fd361ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.491+0000 7fd361ffb640 10 client.?.objecter ms_handle_connect 0x7fd36c1430c0 2024-03-25T18:17:28.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.491+0000 7fd361ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.493+0000 7fd361ffb640 10 client.?.objecter ms_handle_connect 0x7fd36c0a98f0 2024-03-25T18:17:28.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.493+0000 7fd361ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532655550 0 0) 0x7fd36c1d8360 con 0x7fd36c13f710 2024-03-25T18:17:28.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.505 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:17:28.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.504+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd350004eb0 con 0x7fd36c13f710 2024-03-25T18:17:28.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727544225 0 0) 0x7fd36c1d7490 con 0x7fd36c0a98f0 2024-03-25T18:17:28.510 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c1d8360 con 0x7fd36c0a98f0 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012537478 0 0) 0x7fd36c0abb90 con 0x7fd36c1430c0 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:17:28.511 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.509+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c1d7490 con 0x7fd36c1430c0 2024-03-25T18:17:28.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45709827 0 0) 0x7fd36c1d7490 con 0x7fd36c1430c0 2024-03-25T18:17:28.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd36c0abb90 con 0x7fd36c1430c0 2024-03-25T18:17:28.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd364001a60 con 0x7fd36c1430c0 2024-03-25T18:17:28.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.524 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.525 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.523+0000 7fd361ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:28.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2194894243 0 0) 0x7fd36c0abb90 con 0x7fd36c1430c0 2024-03-25T18:17:28.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 >> v1:172.21.15.164:6789/0 conn(0x7fd36c13f710 legacy=0x7fd36c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 >> v1:172.21.15.46:6789/0 conn(0x7fd36c0a98f0 legacy=0x7fd36c138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:28.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 1 monclient: found mon.c 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c1378a0 con 0x7fd36c1430c0 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd361ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd37379a640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd37379a640 10 monclient: _renew_subs 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd37379a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.533+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36c135040 con 0x7fd36c1430c0 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.534+0000 7fd37379a640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:28.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.534+0000 7fd37379a640 10 monclient: _renew_subs 2024-03-25T18:17:28.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.534+0000 7fd37379a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:28.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.534+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36c134620 con 0x7fd36c1430c0 2024-03-25T18:17:28.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.534+0000 7fd37379a640 1 librados: init done 2024-03-25T18:17:28.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.535+0000 7fd37379a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.537 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.535+0000 7fd37379a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.535+0000 7fd37379a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:28.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.535+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36c1d7aa0 con 0x7fd36c1430c0 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd364001f90 con 0x7fd36c1430c0 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd3637fe640 4 set_mon_vals no callback set 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fd364003b10 con 0x7fd36c1430c0 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:28.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.840+0000 7fd361ffb640 10 monclient: dump: 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:28.842 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:28.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.841+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fd364019ae0 con 0x7fd36c1430c0 2024-03-25T18:17:28.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.841+0000 7fd371536640 10 monclient: get_auth_request con 0x7fd35006ddd0 auth_method 0 2024-03-25T18:17:28.843 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.842+0000 7fd361ffb640 10 client.4628.objecter ms_handle_connect 0x7fd35006ddd0 2024-03-25T18:17:28.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fd3640537f0 con 0x7fd36c1430c0 2024-03-25T18:17:28.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 10 client.4628.objecter ms_dispatch 0x7fd36c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:28.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:28.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:28.850 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:17:28.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd36404f130 con 0x7fd36c1430c0 2024-03-25T18:17:28.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.851 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:28.849+0000 7fd361ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:29.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.079+0000 7fd37379a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.080+0000 7fd37379a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.080+0000 7fd37379a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:29.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.080+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd36c0627b0 con 0x7fd36c1430c0 2024-03-25T18:17:29.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.116+0000 7fd361ffb640 1 -- 172.21.15.46:0/1036354528 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fd364051230 con 0x7fd36c1430c0 2024-03-25T18:17:29.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.116+0000 7fd361ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.116+0000 7fd361ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:29.118 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:29.118 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.118+0000 7fd37379a640 10 librados: watch_flush enter 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.118+0000 7fd37379a640 10 librados: watch_flush exit 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.118+0000 7fd37379a640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.118+0000 7fd37379a640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 >> v1:172.21.15.164:6800/1612084612 conn(0x7fd35006ddd0 legacy=0x7fd350070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 10 monclient: shutdown 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 >> v1:172.21.15.46:6790/0 conn(0x7fd36c1430c0 legacy=0x7fd36c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 shutdown_connections 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 >> 172.21.15.46:0/1036354528 conn(0x7fd36c0b7ea0 msgr2=0x7fd36c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 shutdown_connections 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 -- 172.21.15.46:0/1036354528 wait complete. 2024-03-25T18:17:29.121 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:29.119+0000 7fd37379a640 1 librados: shutdown 2024-03-25T18:17:29.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:35.940 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.053+0000 7fb93f84a640 10 monclient: get_monmap_and_config 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.053+0000 7fb93f84a640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.053+0000 7fb93f84a640 10 monclient: monmap: 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:36.053855+0000 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:36.053855+0000 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.054 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:36.055 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:36.055 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:36.055 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.053+0000 7fb93f84a640 1 Processor -- start 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 1 -- start start 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient: init 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9380a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fb93813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:36.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9381430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:36.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb938142520 con 0x7fb9381430c0 2024-03-25T18:17:36.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9380b28a0 con 0x7fb93813f710 2024-03-25T18:17:36.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.054+0000 7fb93f84a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9380b2a30 con 0x7fb9380a98f0 2024-03-25T18:17:36.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.055+0000 7fb93f84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.057 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.055+0000 7fb93f84a640 10 monclient(hunting): authenticate will time out at 7765.316566s 2024-03-25T18:17:36.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.067+0000 7fb93d5e6640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fb9380a98f0 0x7fb9380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:46468/0 (socket says 172.21.15.46:46468) 2024-03-25T18:17:36.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.067+0000 7fb93d5e6640 1 -- 172.21.15.46:0/399319352 learned_addr learned my addr 172.21.15.46:0/399319352 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109507457 0 0) 0x7fb938142520 con 0x7fb9381430c0 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:17:36.078 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.077+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb924003520 con 0x7fb9381430c0 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597854702 0 0) 0x7fb9380b28a0 con 0x7fb93813f710 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:17:36.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.085+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938142520 con 0x7fb93813f710 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167194697 0 0) 0x7fb9380b2a30 con 0x7fb9380a98f0 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:17:36.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.088+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9380b28a0 con 0x7fb9380a98f0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278043483 0 0) 0x7fb924003520 con 0x7fb9381430c0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 >> v1:172.21.15.46:6790/0 conn(0x7fb9380a98f0 legacy=0x7fb9380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 >> v1:172.21.15.164:6789/0 conn(0x7fb93813f710 legacy=0x7fb9381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 monclient: found mon.noname-a 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380b44c0 con 0x7fb9381430c0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb930001a10 con 0x7fb9381430c0 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb92ffff640 10 monclient: dump: 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb93f84a640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:17:36.100 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.098+0000 7fb93f84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:36.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb930001b70 con 0x7fb9381430c0 2024-03-25T18:17:36.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:36.109 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 1 -- 172.21.15.46:0/399319352 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb930001f80 con 0x7fb9381430c0 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb92ffff640 10 monclient: dump: 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb93f84a640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:36.110 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.108+0000 7fb93f84a640 4 set_mon_vals no callback set 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 10 monclient: shutdown 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 1 -- 172.21.15.46:0/399319352 >> v1:172.21.15.46:6789/0 conn(0x7fb9381430c0 legacy=0x7fb93814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 1 -- 172.21.15.46:0/399319352 shutdown_connections 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 1 -- 172.21.15.46:0/399319352 >> 172.21.15.46:0/399319352 conn(0x7fb9380b7ea0 msgr2=0x7fb9380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 1 -- 172.21.15.46:0/399319352 shutdown_connections 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.109+0000 7fb93f84a640 1 -- 172.21.15.46:0/399319352 wait complete. 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.110+0000 7fb93f84a640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.110+0000 7fb93f84a640 10 monclient: monmap: 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:36.112 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 librados: starting msgr at 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 librados: starting objecter 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 Processor -- start 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 -- start start 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 librados: setting wanted keys 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 1 librados: calling monclient init 2024-03-25T18:17:36.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.111+0000 7fb93f84a640 10 monclient: init 2024-03-25T18:17:36.114 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.114 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:36.114 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9380a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:36.114 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-a con 0x7fb93813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9381430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93813b410 con 0x7fb93813f710 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93813b010 con 0x7fb9380a98f0 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb938135230 con 0x7fb9381430c0 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.112+0000 7fb93f84a640 10 monclient(hunting): authenticate will time out at 7765.373796s 2024-03-25T18:17:36.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.115+0000 7fb93dde7640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fb9381430c0 0x7fb9381db2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:46494/0 (socket says 172.21.15.46:46494) 2024-03-25T18:17:36.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.115+0000 7fb93dde7640 1 -- 172.21.15.46:0/1726153158 learned_addr learned my addr 172.21.15.46:0/1726153158 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:36.121 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.120+0000 7fb92dffb640 10 client.?.objecter ms_handle_connect 0x7fb9381430c0 2024-03-25T18:17:36.121 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.120+0000 7fb92dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.123 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.122+0000 7fb92dffb640 10 client.?.objecter ms_handle_connect 0x7fb9380a98f0 2024-03-25T18:17:36.123 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.122+0000 7fb92dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.124+0000 7fb92dffb640 10 client.?.objecter ms_handle_connect 0x7fb93813f710 2024-03-25T18:17:36.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.124+0000 7fb92dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039658019 0 0) 0x7fb93813b010 con 0x7fb9380a98f0 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:17:36.132 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.131+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb91c004eb0 con 0x7fb9380a98f0 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968156408 0 0) 0x7fb93813b410 con 0x7fb93813f710 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:17:36.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.137+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93813b010 con 0x7fb93813f710 2024-03-25T18:17:36.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313358112 0 0) 0x7fb938135230 con 0x7fb9381430c0 2024-03-25T18:17:36.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.141 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:17:36.142 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.140+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93813b410 con 0x7fb9381430c0 2024-03-25T18:17:36.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.158+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525856607 0 0) 0x7fb93813b410 con 0x7fb9381430c0 2024-03-25T18:17:36.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb938135230 con 0x7fb9381430c0 2024-03-25T18:17:36.159 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb930001e80 con 0x7fb9381430c0 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 10 monclient(hunting): dump: 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.159+0000 7fb92dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:36.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507421495 0 0) 0x7fb91c004eb0 con 0x7fb9380a98f0 2024-03-25T18:17:36.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb93813b410 con 0x7fb9380a98f0 2024-03-25T18:17:36.166 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb9340018f0 con 0x7fb9380a98f0 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.165+0000 7fb92dffb640 10 monclient(hunting): dump: 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.167 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2461728754 0 0) 0x7fb93813b010 con 0x7fb93813f710 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb91c004eb0 con 0x7fb93813f710 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb928001970 con 0x7fb93813f710 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:36.168 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.167+0000 7fb92dffb640 10 monclient(hunting): dump: 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.169 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2873194358 0 0) 0x7fb91c004eb0 con 0x7fb93813f710 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 >> v1:172.21.15.46:6790/0 conn(0x7fb9381430c0 legacy=0x7fb9381db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 >> v1:172.21.15.164:6789/0 conn(0x7fb9380a98f0 legacy=0x7fb938130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 1 monclient: found mon.a 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.176 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9381353c0 con 0x7fb93813f710 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb92dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 10 monclient: _renew_subs 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9380b78e0 con 0x7fb93813f710 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 10 monclient: _renew_subs 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9381315f0 con 0x7fb93813f710 2024-03-25T18:17:36.177 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.175+0000 7fb93f84a640 1 librados: init done 2024-03-25T18:17:36.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.176+0000 7fb93f84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.176+0000 7fb93f84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.176+0000 7fb93f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.178 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.176+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9380627b0 con 0x7fb93813f710 2024-03-25T18:17:36.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9280035f0 con 0x7fb93813f710 2024-03-25T18:17:36.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:36.497 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb928003900 con 0x7fb93813f710 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92dffb640 10 monclient: dump: 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:36.498 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:36.499 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:36.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.496+0000 7fb92f7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:36.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.497+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb9280198d0 con 0x7fb93813f710 2024-03-25T18:17:36.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.498+0000 7fb93d5e6640 10 monclient: get_auth_request con 0x7fb91c06e160 auth_method 0 2024-03-25T18:17:36.499 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.498+0000 7fb92dffb640 10 client.4617.objecter ms_handle_connect 0x7fb91c06e160 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb928053520 con 0x7fb93813f710 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 10 client.4617.objecter ms_dispatch 0x7fb938000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb92804ee60 con 0x7fb93813f710 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.505+0000 7fb92dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:36.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.736+0000 7fb93f84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.736+0000 7fb93f84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.736+0000 7fb93f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:36.737 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.736+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb93813b5c0 con 0x7fb93813f710 2024-03-25T18:17:36.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.772+0000 7fb92dffb640 1 -- 172.21.15.46:0/1726153158 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb93813b5c0 con 0x7fb93813f710 2024-03-25T18:17:36.773 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.773+0000 7fb92dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.774 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.773+0000 7fb92dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:36.774 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:36.774 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 10 librados: watch_flush enter 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 10 librados: watch_flush exit 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 10 client.4617.objecter successfully canceled tick 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb91c06e160 legacy=0x7fb91c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 10 monclient: shutdown 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.775+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 >> v1:172.21.15.46:6789/0 conn(0x7fb93813f710 legacy=0x7fb9381dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.776+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 shutdown_connections 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.776+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 >> 172.21.15.46:0/1726153158 conn(0x7fb9380b7ea0 msgr2=0x7fb938074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:36.777 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.776+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 shutdown_connections 2024-03-25T18:17:36.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.776+0000 7fb93f84a640 1 -- 172.21.15.46:0/1726153158 wait complete. 2024-03-25T18:17:36.778 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:36.776+0000 7fb93f84a640 1 librados: shutdown 2024-03-25T18:17:36.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:43.594 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:43.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.708+0000 7f1a49182640 10 monclient: get_monmap_and_config 2024-03-25T18:17:43.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.708+0000 7f1a49182640 10 monclient: build_initial_monmap 2024-03-25T18:17:43.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.708+0000 7f1a49182640 10 monclient: monmap: 2024-03-25T18:17:43.709 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:43.709370+0000 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:43.709370+0000 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:43.710 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:43.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 1 Processor -- start 2024-03-25T18:17:43.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 1 -- start start 2024-03-25T18:17:43.711 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 10 monclient: init 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-a con 0x7f1a440a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.709+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-c con 0x7f1a4413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-b con 0x7f1a441430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a44142520 con 0x7f1a440a98f0 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a440b28a0 con 0x7f1a4413f710 2024-03-25T18:17:43.712 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a440b2a30 con 0x7f1a441430c0 2024-03-25T18:17:43.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:43.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.710+0000 7f1a49182640 10 monclient(hunting): authenticate will time out at 7772.971672s 2024-03-25T18:17:43.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.721+0000 7f1a4359e640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f1a441430c0 0x7f1a4414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:43796/0 (socket says 172.21.15.46:43796) 2024-03-25T18:17:43.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.721+0000 7f1a4359e640 1 -- 172.21.15.46:0/388113078 learned_addr learned my addr 172.21.15.46:0/388113078 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963027783 0 0) 0x7f1a440b28a0 con 0x7f1a4413f710 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:17:43.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.741+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c003520 con 0x7f1a4413f710 2024-03-25T18:17:43.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283444490 0 0) 0x7f1a440b2a30 con 0x7f1a441430c0 2024-03-25T18:17:43.750 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:17:43.751 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.750+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c003840 con 0x7f1a441430c0 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887684107 0 0) 0x7f1a44142520 con 0x7f1a440a98f0 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:17:43.761 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.760+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a440b2a30 con 0x7f1a440a98f0 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1460230690 0 0) 0x7f1a2c003840 con 0x7f1a441430c0 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 >> v1:172.21.15.164:6789/0 conn(0x7f1a4413f710 legacy=0x7f1a441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 >> v1:172.21.15.46:6789/0 conn(0x7f1a440a98f0 legacy=0x7f1a440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 monclient: found mon.noname-b 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a440b44c0 con 0x7f1a441430c0 2024-03-25T18:17:43.766 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1a3c0019f0 con 0x7f1a441430c0 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a41d9b640 10 monclient: dump: 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:43.767 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:43.768 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:43.768 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:43.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a49182640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:17:43.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.765+0000 7f1a49182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:43.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a3c001b50 con 0x7f1a441430c0 2024-03-25T18:17:43.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:43.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 1 -- 172.21.15.46:0/388113078 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1a3c001f60 con 0x7f1a441430c0 2024-03-25T18:17:43.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a49182640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a49182640 4 set_mon_vals no callback set 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.785+0000 7f1a41d9b640 10 monclient: dump: 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:43.787 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 10 monclient: shutdown 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 1 -- 172.21.15.46:0/388113078 >> v1:172.21.15.46:6790/0 conn(0x7f1a441430c0 legacy=0x7f1a4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 1 -- 172.21.15.46:0/388113078 shutdown_connections 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 1 -- 172.21.15.46:0/388113078 >> 172.21.15.46:0/388113078 conn(0x7f1a440b7ea0 msgr2=0x7f1a440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 1 -- 172.21.15.46:0/388113078 shutdown_connections 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.786+0000 7f1a49182640 1 -- 172.21.15.46:0/388113078 wait complete. 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.787+0000 7f1a49182640 10 monclient: build_initial_monmap 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.787+0000 7f1a49182640 10 monclient: monmap: 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:43.789 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 librados: starting msgr at 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 librados: starting objecter 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 Processor -- start 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 -- start start 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 librados: setting wanted keys 2024-03-25T18:17:43.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 1 librados: calling monclient init 2024-03-25T18:17:43.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 10 monclient: init 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.788+0000 7f1a49182640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-c con 0x7f1a440a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-a con 0x7f1a4413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient(hunting): picked mon.noname-b con 0x7f1a441430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a44138d10 con 0x7f1a4413f710 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a441d77a0 con 0x7f1a441430c0 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a44139990 con 0x7f1a440a98f0 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:43.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.789+0000 7f1a49182640 10 monclient(hunting): authenticate will time out at 7773.050702s 2024-03-25T18:17:43.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.801+0000 7f1a42d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f1a440a98f0 0x7f1a44138600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:43892/0 (socket says 172.21.15.46:43892) 2024-03-25T18:17:43.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.801+0000 7f1a42d9d640 1 -- 172.21.15.46:0/1753772869 learned_addr learned my addr 172.21.15.46:0/1753772869 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:43.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.802+0000 7f1a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1a440a98f0 2024-03-25T18:17:43.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.802+0000 7f1a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.803+0000 7f1a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1a441430c0 2024-03-25T18:17:43.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.803+0000 7f1a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914805085 0 0) 0x7f1a44139990 con 0x7f1a440a98f0 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:17:43.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.811+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a30004eb0 con 0x7f1a440a98f0 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042185697 0 0) 0x7f1a441d77a0 con 0x7f1a441430c0 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:17:43.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.813+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a44139990 con 0x7f1a441430c0 2024-03-25T18:17:43.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.818+0000 7f1a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1a4413f710 2024-03-25T18:17:43.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.818+0000 7f1a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806126965 0 0) 0x7f1a30004eb0 con 0x7f1a440a98f0 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a441d77a0 con 0x7f1a440a98f0 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1a34001a30 con 0x7f1a440a98f0 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:43.821 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:43.822 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:43.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.820+0000 7f1a2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2724374213 0 0) 0x7f1a441d77a0 con 0x7f1a440a98f0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 >> v1:172.21.15.164:6789/0 conn(0x7f1a441430c0 legacy=0x7f1a441e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 >> v1:172.21.15.46:6789/0 conn(0x7f1a4413f710 legacy=0x7f1a441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 1 monclient: found mon.c 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a441d8340 con 0x7f1a440a98f0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:43.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a49182640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a49182640 10 monclient: _renew_subs 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a49182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.826+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a441378f0 con 0x7f1a440a98f0 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 monclient: _renew_subs 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a44137c40 con 0x7f1a440a98f0 2024-03-25T18:17:43.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 1 librados: init done 2024-03-25T18:17:43.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:43.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:43.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:43.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:43.827+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a440abd30 con 0x7f1a440a98f0 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a34001c70 con 0x7f1a440a98f0 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1a34003a60 con 0x7f1a440a98f0 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a2b7fe640 10 monclient: dump: 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:44.154 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:44.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.152+0000 7f1a4159a640 4 set_mon_vals no callback set 2024-03-25T18:17:44.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.153+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f1a34003c80 con 0x7f1a440a98f0 2024-03-25T18:17:44.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.154+0000 7f1a4259c640 10 monclient: get_auth_request con 0x7f1a3006dd10 auth_method 0 2024-03-25T18:17:44.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.154+0000 7f1a2b7fe640 10 client.4640.objecter ms_handle_connect 0x7f1a3006dd10 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f1a34054870 con 0x7f1a440a98f0 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 10 client.4640.objecter ms_dispatch 0x7f1a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a34050380 con 0x7f1a440a98f0 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.160 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.159+0000 7f1a2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:44.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.385+0000 7f1a49182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.385+0000 7f1a49182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.385+0000 7f1a49182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:44.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.385+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a441d7500 con 0x7f1a440a98f0 2024-03-25T18:17:44.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.413+0000 7f1a2b7fe640 1 -- 172.21.15.46:0/1753772869 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f1a340522b0 con 0x7f1a440a98f0 2024-03-25T18:17:44.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.413+0000 7f1a2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.413+0000 7f1a2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:44.414 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:44.414 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:44.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 10 librados: watch_flush enter 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 10 librados: watch_flush exit 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 10 client.4640.objecter successfully canceled tick 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 >> v1:172.21.15.164:6800/1612084612 conn(0x7f1a3006dd10 legacy=0x7f1a300701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 10 monclient: shutdown 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 >> v1:172.21.15.46:6790/0 conn(0x7f1a440a98f0 legacy=0x7f1a44138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 shutdown_connections 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 >> 172.21.15.46:0/1753772869 conn(0x7f1a440b7ea0 msgr2=0x7f1a4413ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 shutdown_connections 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 -- 172.21.15.46:0/1753772869 wait complete. 2024-03-25T18:17:44.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:44.416+0000 7f1a49182640 1 librados: shutdown 2024-03-25T18:17:44.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:45.689 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:21.607 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.690 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:21.560 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.690 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:21.561 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.698 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:21.616 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.731 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:21.602 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.751 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:21.668 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.754 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:21.672 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.766 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:21.684 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.799 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:21.716 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:51.236 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:51.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.349+0000 7fa9b5c9a640 10 monclient: get_monmap_and_config 2024-03-25T18:17:51.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.349+0000 7fa9b5c9a640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.349+0000 7fa9b5c9a640 10 monclient: monmap: 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:51.350534+0000 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:51.350534+0000 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:51.351 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:51.352 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.350+0000 7fa9b5c9a640 1 Processor -- start 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.350+0000 7fa9b5c9a640 1 -- start start 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.350+0000 7fa9b5c9a640 10 monclient: init 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fa9b00a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b01430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b0142520 con 0x7fa9b013f710 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00b28a0 con 0x7fa9b00a98f0 2024-03-25T18:17:51.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00b2a30 con 0x7fa9b01430c0 2024-03-25T18:17:51.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.351+0000 7fa9b5c9a640 10 monclient(hunting): authenticate will time out at 7780.613435s 2024-03-25T18:17:51.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.357+0000 7fa9aeffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9b013f710 0x7fa9b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41590/0 (socket says 172.21.15.46:41590) 2024-03-25T18:17:51.358 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.357+0000 7fa9aeffd640 1 -- 172.21.15.46:0/2752506405 learned_addr learned my addr 172.21.15.46:0/2752506405 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:51.376 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.375+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573287089 0 0) 0x7fa9b0142520 con 0x7fa9b013f710 2024-03-25T18:17:51.376 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.375+0000 7fa9ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.375+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.375+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.375+0000 7fa9ae7fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:17:51.377 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.376+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa998003520 con 0x7fa9b013f710 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027019472 0 0) 0x7fa9b00b2a30 con 0x7fa9b01430c0 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:17:51.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.380+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b0142520 con 0x7fa9b01430c0 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081165179 0 0) 0x7fa9b00b28a0 con 0x7fa9b00a98f0 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:17:51.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.381+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b00b2a30 con 0x7fa9b00a98f0 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581892452 0 0) 0x7fa998003520 con 0x7fa9b013f710 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 >> v1:172.21.15.46:6790/0 conn(0x7fa9b01430c0 legacy=0x7fa9b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 >> v1:172.21.15.164:6789/0 conn(0x7fa9b00a98f0 legacy=0x7fa9b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 1 monclient: found mon.noname-a 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00b44c0 con 0x7fa9b013f710 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9b5c9a640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.385+0000 7fa9b5c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.388+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa9a4001650 con 0x7fa9b013f710 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.388+0000 7fa9ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.388+0000 7fa9ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.388+0000 7fa9ae7fc640 10 monclient: dump: 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.389 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.388+0000 7fa9b5c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9a40018a0 con 0x7fa9b013f710 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 1 -- 172.21.15.46:0/2752506405 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa9a4001c30 con 0x7fa9b013f710 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9ae7fc640 10 monclient: dump: 2024-03-25T18:17:51.406 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9b5c9a640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:51.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.405+0000 7fa9b5c9a640 4 set_mon_vals no callback set 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 10 monclient: shutdown 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/2752506405 >> v1:172.21.15.46:6789/0 conn(0x7fa9b013f710 legacy=0x7fa9b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/2752506405 shutdown_connections 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/2752506405 >> 172.21.15.46:0/2752506405 conn(0x7fa9b00b7ea0 msgr2=0x7fa9b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/2752506405 shutdown_connections 2024-03-25T18:17:51.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/2752506405 wait complete. 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.406+0000 7fa9b5c9a640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 10 monclient: monmap: 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:51.411 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 librados: starting msgr at 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 librados: starting objecter 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 Processor -- start 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 -- start start 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 librados: setting wanted keys 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 1 librados: calling monclient init 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.407+0000 7fa9b5c9a640 10 monclient: init 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fa9b00a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b01430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:51.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b0137260 con 0x7fa9b01430c0 2024-03-25T18:17:51.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b01373f0 con 0x7fa9b013f710 2024-03-25T18:17:51.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b0137600 con 0x7fa9b00a98f0 2024-03-25T18:17:51.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.408+0000 7fa9b5c9a640 10 monclient(hunting): authenticate will time out at 7780.669938s 2024-03-25T18:17:51.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.422+0000 7fa9affff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9b01430c0 0x7fa9b01e2e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41626/0 (socket says 172.21.15.46:41626) 2024-03-25T18:17:51.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.422+0000 7fa9affff640 1 -- 172.21.15.46:0/3201279212 learned_addr learned my addr 172.21.15.46:0/3201279212 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:51.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.423+0000 7fa997fff640 10 client.?.objecter ms_handle_connect 0x7fa9b013f710 2024-03-25T18:17:51.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.424+0000 7fa997fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.426+0000 7fa997fff640 10 client.?.objecter ms_handle_connect 0x7fa9b01430c0 2024-03-25T18:17:51.427 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.426+0000 7fa997fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386950495 0 0) 0x7fa9b0137260 con 0x7fa9b01430c0 2024-03-25T18:17:51.435 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:17:51.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.434+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c004eb0 con 0x7fa9b01430c0 2024-03-25T18:17:51.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.437+0000 7fa997fff640 10 client.?.objecter ms_handle_connect 0x7fa9b00a98f0 2024-03-25T18:17:51.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.437+0000 7fa997fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996461838 0 0) 0x7fa9b00b44c0 con 0x7fa9b013f710 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:17:51.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.447+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b0137260 con 0x7fa9b013f710 2024-03-25T18:17:51.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.449+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1535935769 0 0) 0x7fa99c004eb0 con 0x7fa9b01430c0 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.449+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b00b44c0 con 0x7fa9b01430c0 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.449+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa9a8001c00 con 0x7fa9b01430c0 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.449+0000 7fa997fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.450+0000 7fa997fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.450+0000 7fa997fff640 10 monclient(hunting): dump: 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.451 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.450+0000 7fa997fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034001615 0 0) 0x7fa9b0137600 con 0x7fa9b00a98f0 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:17:51.453 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.452+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c004eb0 con 0x7fa9b00a98f0 2024-03-25T18:17:51.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2330442897 0 0) 0x7fa9b00b44c0 con 0x7fa9b01430c0 2024-03-25T18:17:51.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 >> v1:172.21.15.46:6790/0 conn(0x7fa9b00a98f0 legacy=0x7fa9b01d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 >> v1:172.21.15.164:6789/0 conn(0x7fa9b013f710 legacy=0x7fa9b0133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:51.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 1 monclient: found mon.a 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0137790 con 0x7fa9b01430c0 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 10 monclient: _renew_subs 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b0137f00 con 0x7fa9b01430c0 2024-03-25T18:17:51.466 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 10 monclient: _renew_subs 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.463+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b01d7ef0 con 0x7fa9b01430c0 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.464+0000 7fa9b5c9a640 1 librados: init done 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.464+0000 7fa9b5c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.464+0000 7fa9b5c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.464+0000 7fa9b5c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:51.467 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.464+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0137ba0 con 0x7fa9b01430c0 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9a80020f0 con 0x7fa9b01430c0 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa9a8003b30 con 0x7fa9b01430c0 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:51.789 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa997fff640 10 monclient: dump: 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.788+0000 7fa9adffb640 4 set_mon_vals no callback set 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.789+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fa9a8003d50 con 0x7fa9b01430c0 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.789+0000 7fa9af7fe640 10 monclient: get_auth_request con 0x7fa99c06de00 auth_method 0 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.789+0000 7fa997fff640 10 client.4629.objecter ms_handle_connect 0x7fa99c06de00 2024-03-25T18:17:51.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fa9a8054a50 con 0x7fa9b01430c0 2024-03-25T18:17:51.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 10 client.4629.objecter ms_dispatch 0x7fa9b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:51.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:51.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:51.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:17:51.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9a8050390 con 0x7fa9b01430c0 2024-03-25T18:17:51.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:51.795+0000 7fa997fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:52.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.039+0000 7fa9b5c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.039+0000 7fa9b5c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.039+0000 7fa9b5c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:17:52.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.039+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9b0138090 con 0x7fa9b01430c0 2024-03-25T18:17:52.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.132+0000 7fa997fff640 1 -- 172.21.15.46:0/3201279212 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fa9a8052490 con 0x7fa9b01430c0 2024-03-25T18:17:52.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.132+0000 7fa997fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.132+0000 7fa997fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:52.134 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:52.134 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:52.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 10 librados: watch_flush enter 2024-03-25T18:17:52.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 10 librados: watch_flush exit 2024-03-25T18:17:52.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:17:52.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:17:52.136 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 >> v1:172.21.15.164:6800/1612084612 conn(0x7fa99c06de00 legacy=0x7fa99c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 10 monclient: shutdown 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 >> v1:172.21.15.46:6789/0 conn(0x7fa9b01430c0 legacy=0x7fa9b01e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 shutdown_connections 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 >> 172.21.15.46:0/3201279212 conn(0x7fa9b00b7ea0 msgr2=0x7fa9b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 shutdown_connections 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 -- 172.21.15.46:0/3201279212 wait complete. 2024-03-25T18:17:52.137 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:52.135+0000 7fa9b5c9a640 1 librados: shutdown 2024-03-25T18:17:52.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:58.955 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:59.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.066+0000 7f0a1b5d6640 10 monclient: get_monmap_and_config 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.066+0000 7f0a1b5d6640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.066+0000 7f0a1b5d6640 10 monclient: monmap: 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:17:59.068533+0000 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:17:59.068533+0000 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:17:59.069 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.067+0000 7f0a1b5d6640 1 Processor -- start 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.067+0000 7f0a1b5d6640 1 -- start start 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.067+0000 7f0a1b5d6640 10 monclient: init 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a140a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:59.071 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a1413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a141430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a14142520 con 0x7f0a1413f710 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140b28a0 con 0x7f0a140a98f0 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140b2a30 con 0x7f0a141430c0 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.072 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.068+0000 7f0a1b5d6640 10 monclient(hunting): authenticate will time out at 7788.331080s 2024-03-25T18:17:59.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.082+0000 7f0a19372640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0a140a98f0 0x7f0a140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:39234/0 (socket says 172.21.15.46:39234) 2024-03-25T18:17:59.084 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.082+0000 7f0a19372640 1 -- 172.21.15.46:0/1279207849 learned_addr learned my addr 172.21.15.46:0/1279207849 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:59.098 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902767965 0 0) 0x7f0a140b28a0 con 0x7f0a140a98f0 2024-03-25T18:17:59.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:17:59.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.097+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a00003520 con 0x7f0a140a98f0 2024-03-25T18:17:59.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.099+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136001644 0 0) 0x7f0a14142520 con 0x7f0a1413f710 2024-03-25T18:17:59.101 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.100+0000 7f0a0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.100+0000 7f0a0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.100+0000 7f0a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.100+0000 7f0a0bfff640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:17:59.102 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.100+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a140b28a0 con 0x7f0a1413f710 2024-03-25T18:17:59.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367933400 0 0) 0x7f0a140b28a0 con 0x7f0a1413f710 2024-03-25T18:17:59.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 >> v1:172.21.15.46:6790/0 conn(0x7f0a141430c0 legacy=0x7f0a1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.111 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 >> v1:172.21.15.164:6789/0 conn(0x7f0a140a98f0 legacy=0x7f0a140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 monclient: found mon.noname-a 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140b44c0 con 0x7f0a1413f710 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a04001670 con 0x7f0a1413f710 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a1b5d6640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a1b5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.112 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a0bfff640 10 monclient: dump: 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.113 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.109+0000 7f0a1b5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a040017d0 con 0x7f0a1413f710 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 1 -- 172.21.15.46:0/1279207849 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a04001ba0 con 0x7f0a1413f710 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a0bfff640 10 monclient: dump: 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.134 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a1b5d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:59.135 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.132+0000 7f0a1b5d6640 4 set_mon_vals no callback set 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 10 monclient: shutdown 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/1279207849 >> v1:172.21.15.46:6789/0 conn(0x7f0a1413f710 legacy=0x7f0a141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/1279207849 shutdown_connections 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/1279207849 >> 172.21.15.46:0/1279207849 conn(0x7f0a140b7ea0 msgr2=0x7f0a140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/1279207849 shutdown_connections 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.133+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/1279207849 wait complete. 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 10 monclient: monmap: 2024-03-25T18:17:59.138 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 librados: starting msgr at 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 librados: starting objecter 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 Processor -- start 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 -- start start 2024-03-25T18:17:59.139 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 librados: setting wanted keys 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 1 librados: calling monclient init 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.134+0000 7f0a1b5d6640 10 monclient: init 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a140a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a1413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a141430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a141d7830 con 0x7f0a1413f710 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a141d7500 con 0x7f0a140a98f0 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a14137970 con 0x7f0a141430c0 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.140 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.135+0000 7f0a1b5d6640 10 monclient(hunting): authenticate will time out at 7788.398015s 2024-03-25T18:17:59.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.147+0000 7f0a19b73640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f0a141430c0 0x7f0a141e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:35170/0 (socket says 172.21.15.46:35170) 2024-03-25T18:17:59.149 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.147+0000 7f0a19b73640 1 -- 172.21.15.46:0/3358622727 learned_addr learned my addr 172.21.15.46:0/3358622727 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:17:59.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.151+0000 7f0a09ffb640 10 client.?.objecter ms_handle_connect 0x7f0a140a98f0 2024-03-25T18:17:59.152 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.151+0000 7f0a09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.155 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.154+0000 7f0a09ffb640 10 client.?.objecter ms_handle_connect 0x7f0a141430c0 2024-03-25T18:17:59.156 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.154+0000 7f0a09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.161+0000 7f0a09ffb640 10 client.?.objecter ms_handle_connect 0x7f0a1413f710 2024-03-25T18:17:59.163 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.161+0000 7f0a09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395255869 0 0) 0x7f0a141d7500 con 0x7f0a140a98f0 2024-03-25T18:17:59.171 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:17:59.172 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.170+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8004eb0 con 0x7f0a140a98f0 2024-03-25T18:17:59.173 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11378453 0 0) 0x7f0a141d7830 con 0x7f0a1413f710 2024-03-25T18:17:59.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:17:59.174 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.172+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a141d7500 con 0x7f0a1413f710 2024-03-25T18:17:59.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53362871 0 0) 0x7f0a14137970 con 0x7f0a141430c0 2024-03-25T18:17:59.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.185 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.185 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:17:59.185 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.182+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a141d7830 con 0x7f0a141430c0 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2154560615 0 0) 0x7f09f8004eb0 con 0x7f0a140a98f0 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a14137970 con 0x7f0a140a98f0 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a10001950 con 0x7f0a140a98f0 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.186 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.187 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.184+0000 7f0a09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:59.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481998095 0 0) 0x7f0a141d7500 con 0x7f0a1413f710 2024-03-25T18:17:59.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09f8004eb0 con 0x7f0a1413f710 2024-03-25T18:17:59.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a04001ae0 con 0x7f0a1413f710 2024-03-25T18:17:59.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.192+0000 7f0a09ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.195 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.193+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 984115031 0 0) 0x7f0a141d7830 con 0x7f0a141430c0 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.194+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a141d7500 con 0x7f0a141430c0 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.194+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a0c001b50 con 0x7f0a141430c0 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.194+0000 7f0a09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.194+0000 7f0a09ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.194+0000 7f0a09ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.196 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.197 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.197 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.197 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.197 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3892908065 0 0) 0x7f0a141d7500 con 0x7f0a141430c0 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 >> v1:172.21.15.164:6789/0 conn(0x7f0a140a98f0 legacy=0x7f0a14138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 >> v1:172.21.15.46:6789/0 conn(0x7f0a1413f710 legacy=0x7f0a141d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 1 monclient: found mon.c 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14137b00 con 0x7f0a141430c0 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a1b5d6640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a1b5d6640 10 monclient: _renew_subs 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a1b5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.199+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a14135080 con 0x7f0a141430c0 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.200+0000 7f0a1b5d6640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.200+0000 7f0a1b5d6640 10 monclient: _renew_subs 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.200+0000 7f0a1b5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.200+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a141e36d0 con 0x7f0a141430c0 2024-03-25T18:17:59.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.200+0000 7f0a1b5d6640 1 librados: init done 2024-03-25T18:17:59.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.201+0000 7f0a1b5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.201+0000 7f0a1b5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.201+0000 7f0a1b5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:59.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.201+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a141d7690 con 0x7f0a141430c0 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a0c002040 con 0x7f0a141430c0 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f0a0c003990 con 0x7f0a141430c0 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a09ffb640 10 monclient: dump: 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.552 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.550+0000 7f0a0b7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.551+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f0a0c003bb0 con 0x7f0a141430c0 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.551+0000 7f0a19372640 10 monclient: get_auth_request con 0x7f09f806e110 auth_method 0 2024-03-25T18:17:59.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.552+0000 7f0a09ffb640 10 client.4652.objecter ms_handle_connect 0x7f09f806e110 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f0a0c054810 con 0x7f0a141430c0 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 10 client.4652.objecter ms_dispatch 0x7f0a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a0c050150 con 0x7f0a141430c0 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.558+0000 7f0a09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:59.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.795+0000 7f0a1b5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.795+0000 7f0a1b5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.795+0000 7f0a1b5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:17:59.797 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.795+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a140627b0 con 0x7f0a141430c0 2024-03-25T18:17:59.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.824+0000 7f0a09ffb640 1 -- 172.21.15.46:0/3358622727 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f0a0c052250 con 0x7f0a141430c0 2024-03-25T18:17:59.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.824+0000 7f0a09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.824+0000 7f0a09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:59.826 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:17:59.826 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.826+0000 7f0a1b5d6640 10 librados: watch_flush enter 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.826+0000 7f0a1b5d6640 10 librados: watch_flush exit 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.826+0000 7f0a1b5d6640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.826+0000 7f0a1b5d6640 10 client.4652.objecter successfully canceled tick 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.826+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 >> v1:172.21.15.164:6800/1612084612 conn(0x7f09f806e110 legacy=0x7f09f80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 10 monclient: shutdown 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 >> v1:172.21.15.46:6790/0 conn(0x7f0a141430c0 legacy=0x7f0a141e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 shutdown_connections 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 >> 172.21.15.46:0/3358622727 conn(0x7f0a140b7ea0 msgr2=0x7f0a14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:59.829 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 shutdown_connections 2024-03-25T18:17:59.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 -- 172.21.15.46:0/3358622727 wait complete. 2024-03-25T18:17:59.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:17:59.827+0000 7f0a1b5d6640 1 librados: shutdown 2024-03-25T18:17:59.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:06.646 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.763+0000 7fc90fc8d640 10 monclient: get_monmap_and_config 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.763+0000 7fc90fc8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.763+0000 7fc90fc8d640 10 monclient: monmap: 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:06.764938+0000 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:06.764938+0000 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.765 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:06.766 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:06.766 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:06.766 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.764+0000 7fc90fc8d640 1 Processor -- start 2024-03-25T18:18:06.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.764+0000 7fc90fc8d640 1 -- start start 2024-03-25T18:18:06.767 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.764+0000 7fc90fc8d640 10 monclient: init 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.764+0000 7fc90fc8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.764+0000 7fc90fc8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-a con 0x7fc9080a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-c con 0x7fc90813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-b con 0x7fc9081430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908142520 con 0x7fc9080a98f0 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080b28a0 con 0x7fc90813f710 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080b2a30 con 0x7fc9081430c0 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:06.768 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.765+0000 7fc90fc8d640 10 monclient(hunting): authenticate will time out at 7796.027642s 2024-03-25T18:18:06.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.777+0000 7fc90da29640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9080a98f0 0x7fc9080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38276/0 (socket says 172.21.15.46:38276) 2024-03-25T18:18:06.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.777+0000 7fc90e22a640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fc9081430c0 0x7fc90814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:37250/0 (socket says 172.21.15.46:37250) 2024-03-25T18:18:06.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.777+0000 7fc90da29640 1 -- 172.21.15.46:0/1064344530 learned_addr learned my addr 172.21.15.46:0/1064344530 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:06.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954926583 0 0) 0x7fc9080b2a30 con 0x7fc9081430c0 2024-03-25T18:18:06.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.791 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:18:06.792 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.789+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4003520 con 0x7fc9081430c0 2024-03-25T18:18:06.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325643013 0 0) 0x7fc9080b28a0 con 0x7fc90813f710 2024-03-25T18:18:06.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.795 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:18:06.796 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.793+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9080b2a30 con 0x7fc90813f710 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913339096 0 0) 0x7fc908142520 con 0x7fc9080a98f0 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:18:06.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.802+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9080b28a0 con 0x7fc9080a98f0 2024-03-25T18:18:06.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341623691 0 0) 0x7fc8f4003520 con 0x7fc9081430c0 2024-03-25T18:18:06.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 >> v1:172.21.15.164:6789/0 conn(0x7fc90813f710 legacy=0x7fc9081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 >> v1:172.21.15.46:6789/0 conn(0x7fc9080a98f0 legacy=0x7fc9080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: _finish_hunting 0 2024-03-25T18:18:06.807 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 monclient: found mon.noname-b 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080b44c0 con 0x7fc9081430c0 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: _finish_auth 0 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: _check_auth_tickets 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc900001a10 con 0x7fc9081430c0 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90fc8d640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90fc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90ca27640 10 monclient: dump: 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.808 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.809 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.805+0000 7fc90fc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc900001b70 con 0x7fc9081430c0 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 1 -- 172.21.15.46:0/1064344530 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc900001f80 con 0x7fc9081430c0 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90fc8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90fc8d640 4 set_mon_vals no callback set 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.810 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.808+0000 7fc90ca27640 10 monclient: dump: 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:06.811 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 10 monclient: shutdown 2024-03-25T18:18:06.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:06.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 1 -- 172.21.15.46:0/1064344530 >> v1:172.21.15.46:6790/0 conn(0x7fc9081430c0 legacy=0x7fc90814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 1 -- 172.21.15.46:0/1064344530 shutdown_connections 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 1 -- 172.21.15.46:0/1064344530 >> 172.21.15.46:0/1064344530 conn(0x7fc9080b7ea0 msgr2=0x7fc9080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 1 -- 172.21.15.46:0/1064344530 shutdown_connections 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.809+0000 7fc90fc8d640 1 -- 172.21.15.46:0/1064344530 wait complete. 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.810+0000 7fc90fc8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.810+0000 7fc90fc8d640 10 monclient: monmap: 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:06.813 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:06.814 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 librados: starting msgr at 2024-03-25T18:18:06.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 librados: starting objecter 2024-03-25T18:18:06.814 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 Processor -- start 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 -- start start 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 librados: setting wanted keys 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 1 librados: calling monclient init 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.811+0000 7fc90fc8d640 10 monclient: init 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-c con 0x7fc9080a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-a con 0x7fc90813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient(hunting): picked mon.noname-b con 0x7fc9081430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:06.815 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9081daca0 con 0x7fc90813f710 2024-03-25T18:18:06.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908138ab0 con 0x7fc9081430c0 2024-03-25T18:18:06.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080aba90 con 0x7fc9080a98f0 2024-03-25T18:18:06.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:06.816 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.812+0000 7fc90fc8d640 10 monclient(hunting): authenticate will time out at 7796.075044s 2024-03-25T18:18:06.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.816+0000 7fc90e22a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc9081430c0 0x7fc9081d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:58786/0 (socket says 172.21.15.46:58786) 2024-03-25T18:18:06.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.816+0000 7fc90e22a640 1 -- 172.21.15.46:0/3782933942 learned_addr learned my addr 172.21.15.46:0/3782933942 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:06.818 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.817+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc9081430c0 2024-03-25T18:18:06.819 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.817+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.824+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc9080a98f0 2024-03-25T18:18:06.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.824+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.828+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155230219 0 0) 0x7fc908138ab0 con 0x7fc9081430c0 2024-03-25T18:18:06.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.828+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.830 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.828+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.831 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.828+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.831 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.828+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:18:06.831 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.829+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc9081430c0 2024-03-25T18:18:06.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.835+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc90813f710 2024-03-25T18:18:06.836 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.835+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812868552 0 0) 0x7fc9080aba90 con 0x7fc9080a98f0 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:18:06.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.844+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc908138ab0 con 0x7fc9080a98f0 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864973632 0 0) 0x7fc8ec004eb0 con 0x7fc9081430c0 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9080aba90 con 0x7fc9081430c0 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc900001ec0 con 0x7fc9081430c0 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:06.852 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.850+0000 7fc8fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483840835 0 0) 0x7fc9081daca0 con 0x7fc90813f710 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:18:06.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.854+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc90813f710 2024-03-25T18:18:06.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 871002646 0 0) 0x7fc908138ab0 con 0x7fc9080a98f0 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9081daca0 con 0x7fc9080a98f0 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc904001870 con 0x7fc9080a98f0 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.864+0000 7fc8fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.867 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:06.868 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 91269359 0 0) 0x7fc9080aba90 con 0x7fc9081430c0 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 >> v1:172.21.15.46:6790/0 conn(0x7fc9080a98f0 legacy=0x7fc9081383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 >> v1:172.21.15.46:6789/0 conn(0x7fc90813f710 legacy=0x7fc9081db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 1 monclient: found mon.b 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:06.873 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9081d41e0 con 0x7fc9081430c0 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc8fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 10 monclient: _renew_subs 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9081db840 con 0x7fc9081430c0 2024-03-25T18:18:06.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:06.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 10 monclient: _renew_subs 2024-03-25T18:18:06.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:06.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc908138f00 con 0x7fc9081430c0 2024-03-25T18:18:06.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.870+0000 7fc90fc8d640 1 librados: init done 2024-03-25T18:18:06.875 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.871+0000 7fc90fc8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:06.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.871+0000 7fc90fc8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:06.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.871+0000 7fc90fc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:06.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.871+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9081dae50 con 0x7fc9081430c0 2024-03-25T18:18:06.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc900004e10 con 0x7fc9081430c0 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc900002100 con 0x7fc9081430c0 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fe7fc640 10 monclient: dump: 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:06.892 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:06.893 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:06.893 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:06.893 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:06.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.890+0000 7fc8fffff640 4 set_mon_vals no callback set 2024-03-25T18:18:06.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.891+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fc90001adf0 con 0x7fc9081430c0 2024-03-25T18:18:06.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.891+0000 7fc90da29640 10 monclient: get_auth_request con 0x7fc8ec06df70 auth_method 0 2024-03-25T18:18:06.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.892+0000 7fc8fe7fc640 10 client.4642.objecter ms_handle_connect 0x7fc8ec06df70 2024-03-25T18:18:06.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.893+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fc900054a60 con 0x7fc9081430c0 2024-03-25T18:18:06.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.893+0000 7fc8fe7fc640 10 client.4642.objecter ms_dispatch 0x7fc908000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:06.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.893+0000 7fc8fe7fc640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:06.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.893+0000 7fc8fe7fc640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:06.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:06.893+0000 7fc8fe7fc640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:18:07.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.159+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9000503a0 con 0x7fc9081430c0 2024-03-25T18:18:07.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.159+0000 7fc8fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.161 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.159+0000 7fc8fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:07.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.393+0000 7fc90fc8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.393+0000 7fc90fc8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.393+0000 7fc90fc8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:07.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.393+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9080627b0 con 0x7fc9081430c0 2024-03-25T18:18:07.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.470+0000 7fc8fe7fc640 1 -- 172.21.15.46:0/3782933942 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fc9000524a0 con 0x7fc9081430c0 2024-03-25T18:18:07.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.470+0000 7fc8fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.472 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.470+0000 7fc8fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:07.472 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:07.472 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:07.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.472+0000 7fc90fc8d640 10 librados: watch_flush enter 2024-03-25T18:18:07.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 10 librados: watch_flush exit 2024-03-25T18:18:07.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:18:07.475 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 >> v1:172.21.15.164:6800/1612084612 conn(0x7fc8ec06df70 legacy=0x7fc8ec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 10 monclient: shutdown 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 >> v1:172.21.15.164:6789/0 conn(0x7fc9081430c0 legacy=0x7fc9081d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 shutdown_connections 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 >> 172.21.15.46:0/3782933942 conn(0x7fc9080b7ea0 msgr2=0x7fc908074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 shutdown_connections 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 -- 172.21.15.46:0/3782933942 wait complete. 2024-03-25T18:18:07.476 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:07.473+0000 7fc90fc8d640 1 librados: shutdown 2024-03-25T18:18:07.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:10.786 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:46.704 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.796 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:46.713 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.823 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:09:46.741 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.838 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:46.756 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.843 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:46.761 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.867 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:46.738 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.872 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:09:46.789 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.872 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:46.743 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:10.898 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:09:46.769 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:14.293 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:14.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.408+0000 7fe2a9bf4640 10 monclient: get_monmap_and_config 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.408+0000 7fe2a9bf4640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.408+0000 7fe2a9bf4640 10 monclient: monmap: 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:14.410414+0000 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:14.410414+0000 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:14.411 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:14.412 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.409+0000 7fe2a9bf4640 1 Processor -- start 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.409+0000 7fe2a9bf4640 1 -- start start 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.409+0000 7fe2a9bf4640 10 monclient: init 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2a40a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a4142520 con 0x7fe2a413f710 2024-03-25T18:18:14.413 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40b28a0 con 0x7fe2a40a98f0 2024-03-25T18:18:14.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40b2a30 con 0x7fe2a41430c0 2024-03-25T18:18:14.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.414 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.410+0000 7fe2a9bf4640 10 monclient(hunting): authenticate will time out at 7803.672802s 2024-03-25T18:18:14.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.416+0000 7fe2a37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe2a40a98f0 0x7fe2a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48858/0 (socket says 172.21.15.46:48858) 2024-03-25T18:18:14.418 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.416+0000 7fe2a37fe640 1 -- 172.21.15.46:0/1298277781 learned_addr learned my addr 172.21.15.46:0/1298277781 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514232476 0 0) 0x7fe2a40b28a0 con 0x7fe2a40a98f0 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:18:14.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.427+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c003520 con 0x7fe2a40a98f0 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610609443 0 0) 0x7fe2a4142520 con 0x7fe2a413f710 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:18:14.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.429+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a40b28a0 con 0x7fe2a413f710 2024-03-25T18:18:14.442 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315791148 0 0) 0x7fe2a40b28a0 con 0x7fe2a413f710 2024-03-25T18:18:14.442 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 >> v1:172.21.15.46:6790/0 conn(0x7fe2a41430c0 legacy=0x7fe2a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:14.442 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 >> v1:172.21.15.164:6789/0 conn(0x7fe2a40a98f0 legacy=0x7fe2a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.442 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 1 monclient: found mon.noname-a 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40b44c0 con 0x7fe2a413f710 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a9bf4640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:18:14.443 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.438+0000 7fe2a9bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:14.444 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.441+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe298001630 con 0x7fe2a413f710 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.441+0000 7fe2a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.441+0000 7fe2a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.441+0000 7fe2a27fc640 10 monclient: dump: 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.445 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.441+0000 7fe2a9bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe298001880 con 0x7fe2a413f710 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 1 -- 172.21.15.46:0/1298277781 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe298001c10 con 0x7fe2a413f710 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a27fc640 10 monclient: dump: 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.454 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a9bf4640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:14.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.452+0000 7fe2a9bf4640 4 set_mon_vals no callback set 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 10 monclient: shutdown 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1298277781 >> v1:172.21.15.46:6789/0 conn(0x7fe2a413f710 legacy=0x7fe2a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1298277781 shutdown_connections 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1298277781 >> 172.21.15.46:0/1298277781 conn(0x7fe2a40b7ea0 msgr2=0x7fe2a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1298277781 shutdown_connections 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.453+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1298277781 wait complete. 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 10 monclient: monmap: 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:14.458 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 librados: starting msgr at 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 librados: starting objecter 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 Processor -- start 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 -- start start 2024-03-25T18:18:14.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 librados: setting wanted keys 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 1 librados: calling monclient init 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.454+0000 7fe2a9bf4640 10 monclient: init 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2a40a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a41430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a41d76f0 con 0x7fe2a41430c0 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a4139640 con 0x7fe2a413f710 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a4139920 con 0x7fe2a40a98f0 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.455+0000 7fe2a9bf4640 10 monclient(hunting): authenticate will time out at 7803.717989s 2024-03-25T18:18:14.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.457+0000 7fe2a2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe2a413f710 0x7fe2a41d7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48894/0 (socket says 172.21.15.46:48894) 2024-03-25T18:18:14.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.457+0000 7fe2a2ffd640 1 -- 172.21.15.46:0/1527798817 learned_addr learned my addr 172.21.15.46:0/1527798817 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:14.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.460+0000 7fe28bfff640 10 client.?.objecter ms_handle_connect 0x7fe2a413f710 2024-03-25T18:18:14.462 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.460+0000 7fe28bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.464+0000 7fe28bfff640 10 client.?.objecter ms_handle_connect 0x7fe2a40a98f0 2024-03-25T18:18:14.465 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.464+0000 7fe28bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178423224 0 0) 0x7fe2a4139640 con 0x7fe2a413f710 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:18:14.471 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.469+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe290004eb0 con 0x7fe2a413f710 2024-03-25T18:18:14.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.478+0000 7fe28bfff640 10 client.?.objecter ms_handle_connect 0x7fe2a41430c0 2024-03-25T18:18:14.479 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.478+0000 7fe28bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286536934 0 0) 0x7fe2a4139920 con 0x7fe2a40a98f0 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:18:14.482 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.480+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a4139640 con 0x7fe2a40a98f0 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4205803042 0 0) 0x7fe290004eb0 con 0x7fe2a413f710 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2a4139920 con 0x7fe2a413f710 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe2980035f0 con 0x7fe2a413f710 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.489 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.490 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.487+0000 7fe28bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220607323 0 0) 0x7fe2a41d76f0 con 0x7fe2a41430c0 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:18:14.491 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.488+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe290004eb0 con 0x7fe2a41430c0 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 115452346 0 0) 0x7fe2a4139640 con 0x7fe2a40a98f0 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2a41d76f0 con 0x7fe2a40a98f0 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe294001950 con 0x7fe2a40a98f0 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:14.492 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.490+0000 7fe28bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.493 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519175971 0 0) 0x7fe290004eb0 con 0x7fe2a41430c0 2024-03-25T18:18:14.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2a4139640 con 0x7fe2a41430c0 2024-03-25T18:18:14.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe29c001a10 con 0x7fe2a41430c0 2024-03-25T18:18:14.502 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.500+0000 7fe28bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914318980 0 0) 0x7fe2a4139920 con 0x7fe2a413f710 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 >> v1:172.21.15.46:6790/0 conn(0x7fe2a40a98f0 legacy=0x7fe2a4135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 >> v1:172.21.15.46:6789/0 conn(0x7fe2a41430c0 legacy=0x7fe2a41e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 1 monclient: found mon.b 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a41d7280 con 0x7fe2a413f710 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe28bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe2a9bf4640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe2a9bf4640 10 monclient: _renew_subs 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe2a9bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.503+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a4138bc0 con 0x7fe2a413f710 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 monclient: _renew_subs 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a4138750 con 0x7fe2a413f710 2024-03-25T18:18:14.507 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 1 librados: init done 2024-03-25T18:18:14.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:14.508 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.504+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a41d78a0 con 0x7fe2a413f710 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe298003a00 con 0x7fe2a413f710 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe298003c70 con 0x7fe2a413f710 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe28bfff640 10 monclient: dump: 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:14.781 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:14.782 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.779+0000 7fe2a1ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:14.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.780+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fe298003e90 con 0x7fe2a413f710 2024-03-25T18:18:14.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.780+0000 7fe2a37fe640 10 monclient: get_auth_request con 0x7fe29006e160 auth_method 0 2024-03-25T18:18:14.783 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.781+0000 7fe28bfff640 10 client.4648.objecter ms_handle_connect 0x7fe29006e160 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fe2980547c0 con 0x7fe2a413f710 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 10 client.4648.objecter ms_dispatch 0x7fe2a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2980502d0 con 0x7fe2a413f710 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:14.785+0000 7fe28bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:15.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.018+0000 7fe2a9bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.018+0000 7fe2a9bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.018+0000 7fe2a9bf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:15.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.018+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2a414f2d0 con 0x7fe2a413f710 2024-03-25T18:18:15.085 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.084+0000 7fe28bfff640 1 -- 172.21.15.46:0/1527798817 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fe298052200 con 0x7fe2a413f710 2024-03-25T18:18:15.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.084+0000 7fe28bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.086 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.084+0000 7fe28bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:15.086 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:15.086 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 10 librados: watch_flush enter 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 10 librados: watch_flush exit 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 10 client.4648.objecter successfully canceled tick 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 >> v1:172.21.15.164:6800/1612084612 conn(0x7fe29006e160 legacy=0x7fe290070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 10 monclient: shutdown 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 >> v1:172.21.15.164:6789/0 conn(0x7fe2a413f710 legacy=0x7fe2a41d7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 shutdown_connections 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 >> 172.21.15.46:0/1527798817 conn(0x7fe2a40b7ea0 msgr2=0x7fe2a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.089 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 shutdown_connections 2024-03-25T18:18:15.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.086+0000 7fe2a9bf4640 1 -- 172.21.15.46:0/1527798817 wait complete. 2024-03-25T18:18:15.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:15.087+0000 7fe2a9bf4640 1 librados: shutdown 2024-03-25T18:18:15.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:21.906 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.013+0000 7f98875e2640 10 monclient: get_monmap_and_config 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.013+0000 7f98875e2640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.013+0000 7f98875e2640 10 monclient: monmap: 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:22.015337+0000 2024-03-25T18:18:22.015 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:22.015337+0000 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:22.016 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.014+0000 7f98875e2640 1 Processor -- start 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.014+0000 7f98875e2640 1 -- start start 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.014+0000 7f98875e2640 10 monclient: init 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-b con 0x7f98800a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-a con 0x7f988013f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-c con 0x7f98801430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9880142520 con 0x7f988013f710 2024-03-25T18:18:22.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98800b28a0 con 0x7f98801430c0 2024-03-25T18:18:22.018 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98800b2a30 con 0x7f98800a98f0 2024-03-25T18:18:22.018 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.018 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.015+0000 7f98875e2640 10 monclient(hunting): authenticate will time out at 7811.278132s 2024-03-25T18:18:22.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.022+0000 7f9885b7f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f98801430c0 0x7f988014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53720/0 (socket says 172.21.15.46:53720) 2024-03-25T18:18:22.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.022+0000 7f9885b7f640 1 -- 172.21.15.46:0/1489314961 learned_addr learned my addr 172.21.15.46:0/1489314961 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963194448 0 0) 0x7f98800b28a0 con 0x7f98801430c0 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:18:22.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.028+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f986c003520 con 0x7f98801430c0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461528026 0 0) 0x7f986c003520 con 0x7f98801430c0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 >> v1:172.21.15.46:6790/0 conn(0x7f98800a98f0 legacy=0x7f98800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 >> v1:172.21.15.46:6789/0 conn(0x7f988013f710 legacy=0x7f98801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 monclient: found mon.noname-c 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98800b44c0 con 0x7f98801430c0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.039 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9878001a10 con 0x7f98801430c0 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f9877fff640 10 monclient: dump: 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:22.040 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.041 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f98875e2640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:18:22.041 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.037+0000 7f98875e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:22.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9878001b70 con 0x7f98801430c0 2024-03-25T18:18:22.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 1 -- 172.21.15.46:0/1489314961 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f9878001f80 con 0x7f98801430c0 2024-03-25T18:18:22.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 4 set_mon_vals no callback set 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f9877fff640 10 monclient: dump: 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:22.047 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 10 monclient: shutdown 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 1 -- 172.21.15.46:0/1489314961 >> v1:172.21.15.164:6789/0 conn(0x7f98801430c0 legacy=0x7f988014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 1 -- 172.21.15.46:0/1489314961 shutdown_connections 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.045+0000 7f98875e2640 1 -- 172.21.15.46:0/1489314961 >> 172.21.15.46:0/1489314961 conn(0x7f98800b7ea0 msgr2=0x7f98800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.046+0000 7f98875e2640 1 -- 172.21.15.46:0/1489314961 shutdown_connections 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.046+0000 7f98875e2640 1 -- 172.21.15.46:0/1489314961 wait complete. 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.046+0000 7f98875e2640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.046+0000 7f98875e2640 10 monclient: monmap: 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.050 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 librados: starting msgr at 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 librados: starting objecter 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 Processor -- start 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 -- start start 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 librados: setting wanted keys 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 1 librados: calling monclient init 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 10 monclient: init 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.047+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-a con 0x7f98800a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-c con 0x7f988013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:22.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 10 monclient(hunting): picked mon.noname-b con 0x7f98801430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:22.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98801d8340 con 0x7f98800a98f0 2024-03-25T18:18:22.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98801d7710 con 0x7f98801430c0 2024-03-25T18:18:22.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9880138e90 con 0x7f988013f710 2024-03-25T18:18:22.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.048+0000 7f98875e2640 10 monclient(hunting): authenticate will time out at 7811.310710s 2024-03-25T18:18:22.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.058+0000 7f9885b7f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f98801430c0 0x7f9880133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:53734/0 (socket says 172.21.15.46:53734) 2024-03-25T18:18:22.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.058+0000 7f9885b7f640 1 -- 172.21.15.46:0/4187252137 learned_addr learned my addr 172.21.15.46:0/4187252137 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.060+0000 7f9875ffb640 10 client.?.objecter ms_handle_connect 0x7f98801430c0 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.060+0000 7f9875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.067+0000 7f9875ffb640 10 client.?.objecter ms_handle_connect 0x7f98800a98f0 2024-03-25T18:18:22.068 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.067+0000 7f9875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.077+0000 7f9875ffb640 10 client.?.objecter ms_handle_connect 0x7f988013f710 2024-03-25T18:18:22.079 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.077+0000 7f9875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284800325 0 0) 0x7f98801d8340 con 0x7f98800a98f0 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:18:22.080 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.078+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9864004eb0 con 0x7f98800a98f0 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792277451 0 0) 0x7f98801d7710 con 0x7f98801430c0 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:18:22.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.079+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98801d8340 con 0x7f98801430c0 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840708309 0 0) 0x7f9864004eb0 con 0x7f98800a98f0 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98801d7710 con 0x7f98800a98f0 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f987c001930 con 0x7f98800a98f0 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:22.096 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:22.097 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:22.097 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.097 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.094+0000 7f9875ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527203 0 0) 0x7f9880138e90 con 0x7f988013f710 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.097+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9864004eb0 con 0x7f988013f710 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3235753638 0 0) 0x7f98801d7710 con 0x7f98800a98f0 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 >> v1:172.21.15.46:6790/0 conn(0x7f988013f710 legacy=0x7f9880138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 >> v1:172.21.15.164:6789/0 conn(0x7f98801430c0 legacy=0x7f9880133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 1 monclient: found mon.a 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880139020 con 0x7f98800a98f0 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f9875ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f98875e2640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f98875e2640 10 monclient: _renew_subs 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f98875e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.103+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9880137f30 con 0x7f98800a98f0 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.104+0000 7f98875e2640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.104+0000 7f98875e2640 10 monclient: _renew_subs 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.104+0000 7f98875e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.104+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98801352b0 con 0x7f98800a98f0 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.104+0000 7f98875e2640 1 librados: init done 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.105+0000 7f98875e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.105+0000 7f98875e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.105+0000 7f98875e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:22.108 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.105+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9880138230 con 0x7f98800a98f0 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f987c003610 con 0x7f98800a98f0 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f987c0039a0 con 0x7f98800a98f0 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f9875ffb640 10 monclient: dump: 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:22.423 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.421+0000 7f98777fe640 4 set_mon_vals no callback set 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.422+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f987c003bc0 con 0x7f98800a98f0 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.422+0000 7f9884b7d640 10 monclient: get_auth_request con 0x7f986406de00 auth_method 0 2024-03-25T18:18:22.424 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.423+0000 7f9875ffb640 10 client.4653.objecter ms_handle_connect 0x7f986406de00 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f987c053610 con 0x7f98800a98f0 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 10 client.4653.objecter ms_dispatch 0x7f9880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f987c04f120 con 0x7f98800a98f0 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.428+0000 7f9875ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:22.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.646+0000 7f98875e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.646+0000 7f98875e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.646+0000 7f98875e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:22.648 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.646+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98801d78a0 con 0x7f98800a98f0 2024-03-25T18:18:22.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.703+0000 7f9875ffb640 1 -- 172.21.15.46:0/4187252137 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f987c051050 con 0x7f98800a98f0 2024-03-25T18:18:22.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.703+0000 7f9875ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.703+0000 7f9875ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:22.705 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:22.706 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 10 librados: watch_flush enter 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 10 librados: watch_flush exit 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 >> v1:172.21.15.164:6800/1612084612 conn(0x7f986406de00 legacy=0x7f98640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 10 monclient: shutdown 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 >> v1:172.21.15.46:6789/0 conn(0x7f98800a98f0 legacy=0x7f98801d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 shutdown_connections 2024-03-25T18:18:22.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 >> 172.21.15.46:0/4187252137 conn(0x7f98800b7ea0 msgr2=0x7f9880074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 shutdown_connections 2024-03-25T18:18:22.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 -- 172.21.15.46:0/4187252137 wait complete. 2024-03-25T18:18:22.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:22.706+0000 7f98875e2640 1 librados: shutdown 2024-03-25T18:18:22.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:29.525 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 10 monclient: get_monmap_and_config 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 10 monclient: build_initial_monmap 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 10 monclient: monmap: 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:29.646908+0000 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:29.646908+0000 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:29.647 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:29.648 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 1 Processor -- start 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 1 -- start start 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.645+0000 7f8a69348640 10 monclient: init 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-b con 0x7f8a640a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-c con 0x7f8a6413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-a con 0x7f8a641430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:29.649 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a64142520 con 0x7f8a641430c0 2024-03-25T18:18:29.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640b28a0 con 0x7f8a6413f710 2024-03-25T18:18:29.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640b2a30 con 0x7f8a640a98f0 2024-03-25T18:18:29.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:29.650 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.646+0000 7f8a69348640 10 monclient(hunting): authenticate will time out at 7818.909172s 2024-03-25T18:18:29.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.653+0000 7f8a62ffd640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f8a640a98f0 0x7f8a640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:57720/0 (socket says 172.21.15.46:57720) 2024-03-25T18:18:29.655 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.653+0000 7f8a62ffd640 1 -- 172.21.15.46:0/2019482609 learned_addr learned my addr 172.21.15.46:0/2019482609 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385747527 0 0) 0x7f8a640b2a30 con 0x7f8a640a98f0 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:18:29.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.677+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a4c003520 con 0x7f8a640a98f0 2024-03-25T18:18:29.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611724694 0 0) 0x7f8a640b28a0 con 0x7f8a6413f710 2024-03-25T18:18:29.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:18:29.683 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.680+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a640b2a30 con 0x7f8a6413f710 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277799741 0 0) 0x7f8a64142520 con 0x7f8a641430c0 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.683+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a640b28a0 con 0x7f8a641430c0 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903001967 0 0) 0x7f8a4c003520 con 0x7f8a640a98f0 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 >> v1:172.21.15.164:6789/0 conn(0x7f8a6413f710 legacy=0x7f8a641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 >> v1:172.21.15.46:6789/0 conn(0x7f8a641430c0 legacy=0x7f8a6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 monclient: found mon.noname-b 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:29.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a640b44c0 con 0x7f8a640a98f0 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8a54001690 con 0x7f8a640a98f0 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a61ffb640 10 monclient: dump: 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:29.697 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:29.698 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:29.698 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:29.698 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:29.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a69348640 5 monclient: authenticate success, global_id 4670 2024-03-25T18:18:29.698 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.694+0000 7f8a69348640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:29.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.698+0000 7f8a61ffb640 1 -- 172.21.15.46:0/2019482609 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a540017f0 con 0x7f8a640a98f0 2024-03-25T18:18:29.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:29.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:29.700 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 4 set_mon_vals no callback set 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 10 monclient: shutdown 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 1 -- 172.21.15.46:0/2019482609 >> v1:172.21.15.46:6790/0 conn(0x7f8a640a98f0 legacy=0x7f8a640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 1 -- 172.21.15.46:0/2019482609 shutdown_connections 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 1 -- 172.21.15.46:0/2019482609 >> 172.21.15.46:0/2019482609 conn(0x7f8a640b7ea0 msgr2=0x7f8a640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 1 -- 172.21.15.46:0/2019482609 shutdown_connections 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.699+0000 7f8a69348640 1 -- 172.21.15.46:0/2019482609 wait complete. 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.700+0000 7f8a69348640 10 monclient: build_initial_monmap 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.700+0000 7f8a69348640 10 monclient: monmap: 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.700+0000 7f8a69348640 1 librados: starting msgr at 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.700+0000 7f8a69348640 1 librados: starting objecter 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 Processor -- start 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 -- start start 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 librados: setting wanted keys 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 librados: calling monclient init 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient: init 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-b con 0x7f8a640a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-c con 0x7f8a6413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient(hunting): picked mon.noname-a con 0x7f8a641430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:29.705 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a64138700 con 0x7f8a641430c0 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a64138930 con 0x7f8a640a98f0 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a64138c10 con 0x7f8a6413f710 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.701+0000 7f8a69348640 10 monclient(hunting): authenticate will time out at 7818.964359s 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.703+0000 7f8a62ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f8a640a98f0 0x7f8a640b70c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:55672/0 (socket says 172.21.15.46:55672) 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.703+0000 7f8a62ffd640 1 -- 172.21.15.46:0/1585724035 learned_addr learned my addr 172.21.15.46:0/1585724035 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.704+0000 7f8a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a640a98f0 2024-03-25T18:18:29.706 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.705+0000 7f8a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.710 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.709+0000 7f8a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a641430c0 2024-03-25T18:18:29.710 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.709+0000 7f8a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808354343 0 0) 0x7f8a64138930 con 0x7f8a640a98f0 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:18:29.713 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.711+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a50004eb0 con 0x7f8a640a98f0 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284352874 0 0) 0x7f8a64138700 con 0x7f8a641430c0 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:18:29.721 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.719+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a64138930 con 0x7f8a641430c0 2024-03-25T18:18:29.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.722+0000 7f8a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a6413f710 2024-03-25T18:18:29.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.722+0000 7f8a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157287333 0 0) 0x7f8a50004eb0 con 0x7f8a640a98f0 2024-03-25T18:18:29.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a64138700 con 0x7f8a640a98f0 2024-03-25T18:18:29.725 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8a54001ce0 con 0x7f8a640a98f0 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:29.726 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.723+0000 7f8a4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889794050 0 0) 0x7f8a64138930 con 0x7f8a641430c0 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a50004eb0 con 0x7f8a641430c0 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8a5c001bd0 con 0x7f8a641430c0 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.728+0000 7f8a4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424517135 0 0) 0x7f8a64138c10 con 0x7f8a6413f710 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:18:29.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.732+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a64138930 con 0x7f8a6413f710 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690943946 0 0) 0x7f8a64138700 con 0x7f8a640a98f0 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 >> v1:172.21.15.46:6790/0 conn(0x7f8a6413f710 legacy=0x7f8a64137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 >> v1:172.21.15.46:6789/0 conn(0x7f8a641430c0 legacy=0x7f8a641e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 1 monclient: found mon.b 2024-03-25T18:18:29.742 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64138ef0 con 0x7f8a640a98f0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 10 monclient: _renew_subs 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a64136b70 con 0x7f8a640a98f0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 10 monclient: _renew_subs 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a64137e80 con 0x7f8a640a98f0 2024-03-25T18:18:29.743 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.740+0000 7f8a69348640 1 librados: init done 2024-03-25T18:18:29.744 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.741+0000 7f8a69348640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:29.744 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.741+0000 7f8a69348640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:29.744 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.741+0000 7f8a69348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:29.744 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:29.741+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a641396d0 con 0x7f8a640a98f0 2024-03-25T18:18:30.013 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a54004a90 con 0x7f8a640a98f0 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f8a54004e60 con 0x7f8a640a98f0 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a4b7fe640 10 monclient: dump: 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:30.014 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.012+0000 7f8a617fa640 4 set_mon_vals no callback set 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.013+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f8a54005080 con 0x7f8a640a98f0 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.013+0000 7f8a627fc640 10 monclient: get_auth_request con 0x7f8a5006df50 auth_method 0 2024-03-25T18:18:30.015 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.014+0000 7f8a4b7fe640 10 client.4660.objecter ms_handle_connect 0x7f8a5006df50 2024-03-25T18:18:30.020 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f8a540549b0 con 0x7f8a640a98f0 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 10 client.4660.objecter ms_dispatch 0x7f8a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a540504c0 con 0x7f8a640a98f0 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.021 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.019+0000 7f8a4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:30.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.240+0000 7f8a69348640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.240+0000 7f8a69348640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.240+0000 7f8a69348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:18:30.242 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.240+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a64139990 con 0x7f8a640a98f0 2024-03-25T18:18:30.335 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.334+0000 7f8a4b7fe640 1 -- 172.21.15.46:0/1585724035 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f8a540523f0 con 0x7f8a640a98f0 2024-03-25T18:18:30.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.334+0000 7f8a4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.336 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.334+0000 7f8a4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:30.336 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:30.336 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:30.338 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 10 librados: watch_flush enter 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 10 librados: watch_flush exit 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 >> v1:172.21.15.164:6800/1612084612 conn(0x7f8a5006df50 legacy=0x7f8a50070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 10 monclient: shutdown 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 >> v1:172.21.15.164:6789/0 conn(0x7f8a640a98f0 legacy=0x7f8a640b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 shutdown_connections 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 >> 172.21.15.46:0/1585724035 conn(0x7f8a640b7ea0 msgr2=0x7f8a6413edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 shutdown_connections 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 -- 172.21.15.46:0/1585724035 wait complete. 2024-03-25T18:18:30.339 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:30.336+0000 7f8a69348640 1 librados: shutdown 2024-03-25T18:18:30.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:30.720 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:06.637 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.730 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:06.647 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.762 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:06.679 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.784 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:06.702 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.792 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:06.663 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.797 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:06.714 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.800 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:06.670 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.823 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:06.740 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.829 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:06.700 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:37.156 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.270+0000 7f78dcfa3640 10 monclient: get_monmap_and_config 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.270+0000 7f78dcfa3640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.270+0000 7f78dcfa3640 10 monclient: monmap: 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:37.270911+0000 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:37.270911+0000 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.271 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.272 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:37.272 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:37.272 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:37.272 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.270+0000 7f78dcfa3640 1 Processor -- start 2024-03-25T18:18:37.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 1 -- start start 2024-03-25T18:18:37.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient: init 2024-03-25T18:18:37.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-c con 0x7f78d80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-a con 0x7f78d813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.271+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-b con 0x7f78d81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.272+0000 7f78dcfa3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d8142520 con 0x7f78d813f710 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.272+0000 7f78dcfa3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80b28a0 con 0x7f78d80a98f0 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.272+0000 7f78dcfa3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80b2a30 con 0x7f78d81430c0 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.272+0000 7f78dcfa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.272+0000 7f78dcfa3640 10 monclient(hunting): authenticate will time out at 7826.533571s 2024-03-25T18:18:37.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.284+0000 7f78d6d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f78d81430c0 0x7f78d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:59740/0 (socket says 172.21.15.46:59740) 2024-03-25T18:18:37.285 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.284+0000 7f78d6d9d640 1 -- 172.21.15.46:0/2541019679 learned_addr learned my addr 172.21.15.46:0/2541019679 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612031785 0 0) 0x7f78d80b2a30 con 0x7f78d81430c0 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.300+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c0003520 con 0x7f78d81430c0 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407132189 0 0) 0x7f78d80b28a0 con 0x7f78d80a98f0 2024-03-25T18:18:37.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:18:37.302 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.301+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d80b2a30 con 0x7f78d80a98f0 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439128274 0 0) 0x7f78d8142520 con 0x7f78d813f710 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.316+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d80b28a0 con 0x7f78d813f710 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652848895 0 0) 0x7f78d80b2a30 con 0x7f78d80a98f0 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 >> v1:172.21.15.46:6790/0 conn(0x7f78d81430c0 legacy=0x7f78d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 >> v1:172.21.15.46:6789/0 conn(0x7f78d813f710 legacy=0x7f78d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 monclient: found mon.noname-c 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80b44c0 con 0x7f78d80a98f0 2024-03-25T18:18:37.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78c8001670 con 0x7f78d80a98f0 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78d559a640 10 monclient: dump: 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.328 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.329 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.329 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78dcfa3640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:18:37.329 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.326+0000 7f78dcfa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78c80017d0 con 0x7f78d80a98f0 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 1 -- 172.21.15.46:0/2541019679 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78c8001be0 con 0x7f78d80a98f0 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78dcfa3640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78dcfa3640 4 set_mon_vals no callback set 2024-03-25T18:18:37.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.349+0000 7f78d559a640 10 monclient: dump: 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 10 monclient: shutdown 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 1 -- 172.21.15.46:0/2541019679 >> v1:172.21.15.164:6789/0 conn(0x7f78d80a98f0 legacy=0x7f78d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 1 -- 172.21.15.46:0/2541019679 shutdown_connections 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 1 -- 172.21.15.46:0/2541019679 >> 172.21.15.46:0/2541019679 conn(0x7f78d80b7ea0 msgr2=0x7f78d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 1 -- 172.21.15.46:0/2541019679 shutdown_connections 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.350+0000 7f78dcfa3640 1 -- 172.21.15.46:0/2541019679 wait complete. 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.351+0000 7f78dcfa3640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.351+0000 7f78dcfa3640 10 monclient: monmap: 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:37.353 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.351+0000 7f78dcfa3640 1 librados: starting msgr at 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.351+0000 7f78dcfa3640 1 librados: starting objecter 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 1 Processor -- start 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 1 -- start start 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 1 librados: setting wanted keys 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 1 librados: calling monclient init 2024-03-25T18:18:37.354 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 10 monclient: init 2024-03-25T18:18:37.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-b con 0x7f78d80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.352+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-a con 0x7f78d813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 10 monclient(hunting): picked mon.noname-c con 0x7f78d81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d8139330 con 0x7f78d813f710 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80abe70 con 0x7f78d80a98f0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80ac150 con 0x7f78d81430c0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.353+0000 7f78dcfa3640 10 monclient(hunting): authenticate will time out at 7826.614761s 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.363+0000 7f78d6d9d640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f78d81430c0 0x7f78d81e2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:59804/0 (socket says 172.21.15.46:59804) 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.363+0000 7f78d6d9d640 1 -- 172.21.15.46:0/3599698854 learned_addr learned my addr 172.21.15.46:0/3599698854 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:37.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.368+0000 7f78c6ffd640 10 client.?.objecter ms_handle_connect 0x7f78d81430c0 2024-03-25T18:18:37.369 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.368+0000 7f78c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.378 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.377+0000 7f78c6ffd640 10 client.?.objecter ms_handle_connect 0x7f78d813f710 2024-03-25T18:18:37.378 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.377+0000 7f78c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088970859 0 0) 0x7f78d80ac150 con 0x7f78d81430c0 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:18:37.393 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.392+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b8004eb0 con 0x7f78d81430c0 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132546564 0 0) 0x7f78d8139330 con 0x7f78d813f710 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:18:37.396 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.395+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d80ac150 con 0x7f78d813f710 2024-03-25T18:18:37.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.397+0000 7f78c6ffd640 10 client.?.objecter ms_handle_connect 0x7f78d80a98f0 2024-03-25T18:18:37.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.397+0000 7f78c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651323178 0 0) 0x7f78d80abe70 con 0x7f78d80a98f0 2024-03-25T18:18:37.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:18:37.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.406+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d8139330 con 0x7f78d80a98f0 2024-03-25T18:18:37.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102088746 0 0) 0x7f78d80ac150 con 0x7f78d813f710 2024-03-25T18:18:37.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78d80abe70 con 0x7f78d813f710 2024-03-25T18:18:37.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78cc0018b0 con 0x7f78d813f710 2024-03-25T18:18:37.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.416 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.417 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.415+0000 7f78c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306050743 0 0) 0x7f78b8004eb0 con 0x7f78d81430c0 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78d80ac150 con 0x7f78d81430c0 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78d0001b30 con 0x7f78d81430c0 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.419+0000 7f78c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.420 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.421 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.421 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.421 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.421 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048077691 0 0) 0x7f78d8139330 con 0x7f78d80a98f0 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78b8004eb0 con 0x7f78d80a98f0 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78c8001ae0 con 0x7f78d80a98f0 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.430+0000 7f78c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.431 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.432 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.446 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1768548406 0 0) 0x7f78d80abe70 con 0x7f78d813f710 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 >> v1:172.21.15.46:6790/0 conn(0x7f78d81430c0 legacy=0x7f78d81e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 >> v1:172.21.15.164:6789/0 conn(0x7f78d80a98f0 legacy=0x7f78d81389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 1 monclient: found mon.a 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80acc80 con 0x7f78d813f710 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78dcfa3640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78dcfa3640 10 monclient: _renew_subs 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78dcfa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:37.447 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.445+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d8137e30 con 0x7f78d813f710 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.446+0000 7f78dcfa3640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.446+0000 7f78dcfa3640 10 monclient: _renew_subs 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.446+0000 7f78dcfa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.446+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d8136880 con 0x7f78d813f710 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.446+0000 7f78dcfa3640 1 librados: init done 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.447+0000 7f78dcfa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.447+0000 7f78dcfa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.447+0000 7f78dcfa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:37.448 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.447+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d8136b30 con 0x7f78d813f710 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78cc003610 con 0x7f78d813f710 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f78cc003920 con 0x7f78d813f710 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 10 monclient: dump: 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.779 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:37.780 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78d4d99640 4 set_mon_vals no callback set 2024-03-25T18:18:37.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.778+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f78cc0198f0 con 0x7f78d813f710 2024-03-25T18:18:37.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.779+0000 7f78d659c640 10 monclient: get_auth_request con 0x7f78b806e0f0 auth_method 0 2024-03-25T18:18:37.781 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.779+0000 7f78c6ffd640 10 client.4665.objecter ms_handle_connect 0x7f78b806e0f0 2024-03-25T18:18:37.785 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7f78cc0535d0 con 0x7f78d813f710 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 10 client.4665.objecter ms_dispatch 0x7f78d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 3 client.4665.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 3 client.4665.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78cc04ef10 con 0x7f78d813f710 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:37.784+0000 7f78c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:38.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.045+0000 7f78dcfa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.045+0000 7f78dcfa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.045+0000 7f78dcfa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:38.049 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.045+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78d80627b0 con 0x7f78d813f710 2024-03-25T18:18:38.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.090+0000 7f78c6ffd640 1 -- 172.21.15.46:0/3599698854 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7f78cc051010 con 0x7f78d813f710 2024-03-25T18:18:38.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.090+0000 7f78c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.091 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.090+0000 7f78c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:38.091 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:38.091 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 10 librados: watch_flush enter 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 10 librados: watch_flush exit 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 >> v1:172.21.15.164:6800/1612084612 conn(0x7f78b806e0f0 legacy=0x7f78b80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 10 monclient: shutdown 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.092+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 >> v1:172.21.15.46:6789/0 conn(0x7f78d813f710 legacy=0x7f78d80ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.093+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 shutdown_connections 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.093+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 >> 172.21.15.46:0/3599698854 conn(0x7f78d80b7ea0 msgr2=0x7f78d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.093+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 shutdown_connections 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.093+0000 7f78dcfa3640 1 -- 172.21.15.46:0/3599698854 wait complete. 2024-03-25T18:18:38.095 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:38.093+0000 7f78dcfa3640 1 librados: shutdown 2024-03-25T18:18:38.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:44.912 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:45.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.007+0000 7fb7b91cf640 10 monclient: get_monmap_and_config 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.007+0000 7fb7b91cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.007+0000 7fb7b91cf640 10 monclient: monmap: 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:45.008634+0000 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:45.008634+0000 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:45.009 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.008+0000 7fb7b91cf640 1 Processor -- start 2024-03-25T18:18:45.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.008+0000 7fb7b91cf640 1 -- start start 2024-03-25T18:18:45.010 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.008+0000 7fb7b91cf640 10 monclient: init 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb7b40a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb7b413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb7b41430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b4142520 con 0x7fb7b40a98f0 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b40b28a0 con 0x7fb7b413f710 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b40b2a30 con 0x7fb7b41430c0 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.011 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.009+0000 7fb7b91cf640 10 monclient(hunting): authenticate will time out at 7834.270934s 2024-03-25T18:18:45.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.017+0000 7fb7b259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb7b413f710 0x7fb7b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:39386/0 (socket says 172.21.15.46:39386) 2024-03-25T18:18:45.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.017+0000 7fb7b259c640 1 -- 172.21.15.46:0/772698534 learned_addr learned my addr 172.21.15.46:0/772698534 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:45.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472641003 0 0) 0x7fb7b40b28a0 con 0x7fb7b413f710 2024-03-25T18:18:45.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:18:45.030 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.029+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb79c003520 con 0x7fb7b413f710 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665580291 0 0) 0x7fb7b40b2a30 con 0x7fb7b41430c0 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:18:45.034 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.033+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b40b28a0 con 0x7fb7b41430c0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.043+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285775827 0 0) 0x7fb7b40b28a0 con 0x7fb7b41430c0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 >> v1:172.21.15.164:6789/0 conn(0x7fb7b413f710 legacy=0x7fb7b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 >> v1:172.21.15.46:6789/0 conn(0x7fb7b40a98f0 legacy=0x7fb7b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b40b44c0 con 0x7fb7b41430c0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.045 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb7ac0019f0 con 0x7fb7b41430c0 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b1d9b640 10 monclient: dump: 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:45.046 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:45.047 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b91cf640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:18:45.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.044+0000 7fb7b91cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ac001b50 con 0x7fb7b41430c0 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 1 -- 172.21.15.46:0/772698534 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb7ac001f20 con 0x7fb7b41430c0 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:45.055 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b1d9b640 10 monclient: dump: 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b91cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:45.056 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.054+0000 7fb7b91cf640 4 set_mon_vals no callback set 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 10 monclient: shutdown 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 1 -- 172.21.15.46:0/772698534 >> v1:172.21.15.46:6790/0 conn(0x7fb7b41430c0 legacy=0x7fb7b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 1 -- 172.21.15.46:0/772698534 shutdown_connections 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 1 -- 172.21.15.46:0/772698534 >> 172.21.15.46:0/772698534 conn(0x7fb7b40b7ea0 msgr2=0x7fb7b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 1 -- 172.21.15.46:0/772698534 shutdown_connections 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.055+0000 7fb7b91cf640 1 -- 172.21.15.46:0/772698534 wait complete. 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.056+0000 7fb7b91cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.056+0000 7fb7b91cf640 10 monclient: monmap: 2024-03-25T18:18:45.058 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 librados: starting msgr at 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 librados: starting objecter 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 Processor -- start 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 -- start start 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 librados: setting wanted keys 2024-03-25T18:18:45.059 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 1 librados: calling monclient init 2024-03-25T18:18:45.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.057+0000 7fb7b91cf640 10 monclient: init 2024-03-25T18:18:45.060 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb7b40a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb7b413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb7b41430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b4137e20 con 0x7fb7b41430c0 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b40b1720 con 0x7fb7b413f710 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b4138710 con 0x7fb7b40a98f0 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.061 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.058+0000 7fb7b91cf640 10 monclient(hunting): authenticate will time out at 7834.320028s 2024-03-25T18:18:45.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.074+0000 7fb7b359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7b41430c0 0x7fb7b41daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48250/0 (socket says 172.21.15.46:48250) 2024-03-25T18:18:45.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.074+0000 7fb7b359e640 1 -- 172.21.15.46:0/3169436487 learned_addr learned my addr 172.21.15.46:0/3169436487 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:45.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.075+0000 7fb79b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7b41430c0 2024-03-25T18:18:45.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.075+0000 7fb79b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.077+0000 7fb79b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7b40a98f0 2024-03-25T18:18:45.077 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.077+0000 7fb79b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.088+0000 7fb79b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7b413f710 2024-03-25T18:18:45.088 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.088+0000 7fb79b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311936479 0 0) 0x7fb7b4138710 con 0x7fb7b40a98f0 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:18:45.090 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.089+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0004eb0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523298373 0 0) 0x7fb7b4137e20 con 0x7fb7b41430c0 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:18:45.099 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.098+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4138710 con 0x7fb7b41430c0 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267203970 0 0) 0x7fb7a0004eb0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7b4137e20 con 0x7fb7b40a98f0 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb7a4001930 con 0x7fb7b40a98f0 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:45.103 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.104 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.102+0000 7fb79b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72701666 0 0) 0x7fb7b40b1720 con 0x7fb7b413f710 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:18:45.107 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.106+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0004eb0 con 0x7fb7b413f710 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.116+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 470791562 0 0) 0x7fb7b4137e20 con 0x7fb7b40a98f0 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.116+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 >> v1:172.21.15.164:6789/0 conn(0x7fb7b413f710 legacy=0x7fb7b41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.116+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 >> v1:172.21.15.46:6789/0 conn(0x7fb7b41430c0 legacy=0x7fb7b41daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.116+0000 7fb79b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.116+0000 7fb79b7fe640 1 monclient: found mon.c 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4138040 con 0x7fb7b40a98f0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb79b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 10 monclient: _renew_subs 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b41382f0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 10 monclient: _renew_subs 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b41e2fb0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.117+0000 7fb7b91cf640 1 librados: init done 2024-03-25T18:18:45.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.118+0000 7fb7b91cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.118+0000 7fb7b91cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.118+0000 7fb7b91cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.118+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b40b3ef0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7a4003610 con 0x7fb7b40a98f0 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fb7a40039a0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb79b7fe640 10 monclient: dump: 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:45.430 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.428+0000 7fb7b159a640 4 set_mon_vals no callback set 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.429+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fb7a4003bc0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.431 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.430+0000 7fb7b259c640 10 monclient: get_auth_request con 0x7fb7a006de00 auth_method 0 2024-03-25T18:18:45.432 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.430+0000 7fb79b7fe640 10 client.4685.objecter ms_handle_connect 0x7fb7a006de00 2024-03-25T18:18:45.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.437+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3469872995 0 0) 0x7fb7a4053620 con 0x7fb7b40a98f0 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.437+0000 7fb79b7fe640 10 client.4685.objecter ms_dispatch 0x7fb7b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.437+0000 7fb79b7fe640 3 client.4685.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.437+0000 7fb79b7fe640 3 client.4685.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.438+0000 7fb79b7fe640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.438+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7a404f130 con 0x7fb7b40a98f0 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.438+0000 7fb79b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.438+0000 7fb79b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:45.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.693+0000 7fb7b91cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.693+0000 7fb7b91cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.693+0000 7fb7b91cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:18:45.694 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.693+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7b40627b0 con 0x7fb7b40a98f0 2024-03-25T18:18:45.745 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.744+0000 7fb79b7fe640 1 -- 172.21.15.46:0/3169436487 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4011779686) 0x7fb7a4051060 con 0x7fb7b40a98f0 2024-03-25T18:18:45.745 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.744+0000 7fb79b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.745 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.744+0000 7fb79b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:45.745 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:45.745 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:08:46.408066+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:08:46.408066+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 10 librados: watch_flush enter 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 10 librados: watch_flush exit 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 10 client.4685.objecter successfully canceled tick 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 >> v1:172.21.15.164:6800/1612084612 conn(0x7fb7a006de00 legacy=0x7fb7a00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 10 monclient: shutdown 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 >> v1:172.21.15.46:6790/0 conn(0x7fb7b40a98f0 legacy=0x7fb7b4137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 shutdown_connections 2024-03-25T18:18:45.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.746+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 >> 172.21.15.46:0/3169436487 conn(0x7fb7b40b7ea0 msgr2=0x7fb7b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:45.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.747+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 shutdown_connections 2024-03-25T18:18:45.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.747+0000 7fb7b91cf640 1 -- 172.21.15.46:0/3169436487 wait complete. 2024-03-25T18:18:45.749 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:45.747+0000 7fb7b91cf640 1 librados: shutdown 2024-03-25T18:18:45.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:50.806 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:26.723 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.828 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:26.746 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.862 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:26.780 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.875 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:26.793 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.879 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:26.796 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.893 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:26.764 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.899 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:26.770 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.915 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:26.833 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:50.919 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:26.790 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.566 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.679+0000 7f1f46e3e640 10 monclient: get_monmap_and_config 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.679+0000 7f1f46e3e640 10 monclient: build_initial_monmap 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.679+0000 7f1f46e3e640 10 monclient: monmap: 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:18:52.680332+0000 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:18:52.680332+0000 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:52.680 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:52.681 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:18:52.681 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:18:52.681 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.680+0000 7f1f46e3e640 1 Processor -- start 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.680+0000 7f1f46e3e640 1 -- start start 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.680+0000 7f1f46e3e640 10 monclient: init 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f400a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4013f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f401430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f40142520 con 0x7f1f401430c0 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b28a0 con 0x7f1f400a98f0 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b2a30 con 0x7f1f4013f710 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:52.682 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.681+0000 7f1f46e3e640 10 monclient(hunting): authenticate will time out at 7841.942673s 2024-03-25T18:18:52.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.692+0000 7f1f3ffff640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f1f4013f710 0x7f1f401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:35494/0 (socket says 172.21.15.46:35494) 2024-03-25T18:18:52.693 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.692+0000 7f1f3ffff640 1 -- 172.21.15.46:0/173538195 learned_addr learned my addr 172.21.15.46:0/173538195 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638466124 0 0) 0x7f1f400b28a0 con 0x7f1f400a98f0 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f28003520 con 0x7f1f400a98f0 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.707+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508499931 0 0) 0x7f1f400b2a30 con 0x7f1f4013f710 2024-03-25T18:18:52.708 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.708+0000 7f1f3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.708+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.708+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.708+0000 7f1f3f7fe640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:18:52.709 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.708+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f400b28a0 con 0x7f1f4013f710 2024-03-25T18:18:52.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038076537 0 0) 0x7f1f40142520 con 0x7f1f401430c0 2024-03-25T18:18:52.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.716 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:18:52.717 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.716+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f400b2a30 con 0x7f1f401430c0 2024-03-25T18:18:52.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 253724473 0 0) 0x7f1f400b28a0 con 0x7f1f4013f710 2024-03-25T18:18:52.722 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 >> v1:172.21.15.164:6789/0 conn(0x7f1f400a98f0 legacy=0x7f1f400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 >> v1:172.21.15.46:6789/0 conn(0x7f1f401430c0 legacy=0x7f1f4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400b44c0 con 0x7f1f4013f710 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f34001670 con 0x7f1f4013f710 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:52.723 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f3f7fe640 10 monclient: dump: 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f46e3e640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:18:52.724 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.721+0000 7f1f46e3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:52.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f340017d0 con 0x7f1f4013f710 2024-03-25T18:18:52.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:52.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 1 -- 172.21.15.46:0/173538195 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f34001be0 con 0x7f1f4013f710 2024-03-25T18:18:52.730 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f46e3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f46e3e640 4 set_mon_vals no callback set 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.729+0000 7f1f3f7fe640 10 monclient: dump: 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:52.731 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:52.732 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 10 monclient: shutdown 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 1 -- 172.21.15.46:0/173538195 >> v1:172.21.15.46:6790/0 conn(0x7f1f4013f710 legacy=0x7f1f401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 1 -- 172.21.15.46:0/173538195 shutdown_connections 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 1 -- 172.21.15.46:0/173538195 >> 172.21.15.46:0/173538195 conn(0x7f1f400b7ea0 msgr2=0x7f1f400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.730+0000 7f1f46e3e640 1 -- 172.21.15.46:0/173538195 shutdown_connections 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.731+0000 7f1f46e3e640 1 -- 172.21.15.46:0/173538195 wait complete. 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.731+0000 7f1f46e3e640 10 monclient: build_initial_monmap 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.731+0000 7f1f46e3e640 10 monclient: monmap: 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:18:52.733 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 librados: starting msgr at 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 librados: starting objecter 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 Processor -- start 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 -- start start 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 librados: setting wanted keys 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 1 librados: calling monclient init 2024-03-25T18:18:52.734 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.732+0000 7f1f46e3e640 10 monclient: init 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f400a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4013f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f401430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:18:52.735 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f40138700 con 0x7f1f400a98f0 2024-03-25T18:18:52.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f40138b90 con 0x7f1f4013f710 2024-03-25T18:18:52.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b16d0 con 0x7f1f401430c0 2024-03-25T18:18:52.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:52.736 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.733+0000 7f1f46e3e640 10 monclient(hunting): authenticate will time out at 7841.995138s 2024-03-25T18:18:52.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.745+0000 7f1f453db640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f1f401430c0 0x7f1f401e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:35536/0 (socket says 172.21.15.46:35536) 2024-03-25T18:18:52.746 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.745+0000 7f1f453db640 1 -- 172.21.15.46:0/4274596663 learned_addr learned my addr 172.21.15.46:0/4274596663 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:18:52.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.747+0000 7f1f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f401430c0 2024-03-25T18:18:52.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.747+0000 7f1f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.748+0000 7f1f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f400a98f0 2024-03-25T18:18:52.748 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.748+0000 7f1f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.755+0000 7f1f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f4013f710 2024-03-25T18:18:52.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.755+0000 7f1f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087427317 0 0) 0x7f1f400b16d0 con 0x7f1f401430c0 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:18:52.760 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.759+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c004eb0 con 0x7f1f401430c0 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740964072 0 0) 0x7f1f40138b90 con 0x7f1f4013f710 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:18:52.770 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.769+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f400b16d0 con 0x7f1f4013f710 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866996945 0 0) 0x7f1f40138700 con 0x7f1f400a98f0 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:18:52.772 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.771+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40138b90 con 0x7f1f400a98f0 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279919611 0 0) 0x7f1f40138b90 con 0x7f1f400a98f0 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f40138700 con 0x7f1f400a98f0 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f300017b0 con 0x7f1f400a98f0 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:52.786 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.785+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2063998244 0 0) 0x7f1f2c004eb0 con 0x7f1f401430c0 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.786+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f40138b90 con 0x7f1f401430c0 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.786+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f38001d30 con 0x7f1f401430c0 2024-03-25T18:18:52.787 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.786+0000 7f1f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.786+0000 7f1f3d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.786+0000 7f1f3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:52.788 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208989029 0 0) 0x7f1f40138700 con 0x7f1f400a98f0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 >> v1:172.21.15.46:6790/0 conn(0x7f1f401430c0 legacy=0x7f1f401e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 >> v1:172.21.15.164:6789/0 conn(0x7f1f4013f710 legacy=0x7f1f401da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 1 monclient: found mon.a 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f40138050 con 0x7f1f400a98f0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f46e3e640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f46e3e640 10 monclient: _renew_subs 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f46e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:52.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.800+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f401382a0 con 0x7f1f400a98f0 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.801+0000 7f1f46e3e640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.801+0000 7f1f46e3e640 10 monclient: _renew_subs 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.801+0000 7f1f46e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.801+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f401db170 con 0x7f1f400a98f0 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.801+0000 7f1f46e3e640 1 librados: init done 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.802+0000 7f1f46e3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.802+0000 7f1f46e3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.802+0000 7f1f46e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:52.804 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:52.802+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f401384e0 con 0x7f1f400a98f0 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f30001cc0 con 0x7f1f400a98f0 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1f300037b0 con 0x7f1f400a98f0 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:53.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3d7fa640 10 monclient: dump: 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:18:53.117 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.115+0000 7f1f3effd640 4 set_mon_vals no callback set 2024-03-25T18:18:53.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.116+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f1f30019780 con 0x7f1f400a98f0 2024-03-25T18:18:53.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.117+0000 7f1f3ffff640 10 monclient: get_auth_request con 0x7f1f2c06def0 auth_method 0 2024-03-25T18:18:53.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.118+0000 7f1f3d7fa640 10 client.4677.objecter ms_handle_connect 0x7f1f2c06def0 2024-03-25T18:18:53.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f1f30053480 con 0x7f1f400a98f0 2024-03-25T18:18:53.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 10 client.4677.objecter ms_dispatch 0x7f1f40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:53.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:53.124 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:53.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:18:53.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f3004edc0 con 0x7f1f400a98f0 2024-03-25T18:18:53.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.125 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.123+0000 7f1f3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:53.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.371+0000 7f1f46e3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.371+0000 7f1f46e3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.371+0000 7f1f46e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:18:53.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.371+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f400b3ef0 con 0x7f1f400a98f0 2024-03-25T18:18:53.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.405+0000 7f1f3d7fa640 1 -- 172.21.15.46:0/4274596663 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f1f30050ec0 con 0x7f1f400a98f0 2024-03-25T18:18:53.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.405+0000 7f1f3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.406 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.405+0000 7f1f3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:53.406 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:18:53.407 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 10 librados: watch_flush enter 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 10 librados: watch_flush exit 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 >> v1:172.21.15.164:6800/1612084612 conn(0x7f1f2c06def0 legacy=0x7f1f2c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 10 monclient: shutdown 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 >> v1:172.21.15.46:6789/0 conn(0x7f1f400a98f0 legacy=0x7f1f40137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 shutdown_connections 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 >> 172.21.15.46:0/4274596663 conn(0x7f1f400b7ea0 msgr2=0x7f1f40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 shutdown_connections 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 -- 172.21.15.46:0/4274596663 wait complete. 2024-03-25T18:18:53.410 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:18:53.408+0000 7f1f46e3e640 1 librados: shutdown 2024-03-25T18:18:53.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:00.230 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:00.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.344+0000 7fc1c0481640 10 monclient: get_monmap_and_config 2024-03-25T18:19:00.345 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.344+0000 7fc1c0481640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.344+0000 7fc1c0481640 10 monclient: monmap: 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:00.345515+0000 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:00.345515+0000 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:00.346 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.347 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.345+0000 7fc1c0481640 1 Processor -- start 2024-03-25T18:19:00.347 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.345+0000 7fc1c0481640 1 -- start start 2024-03-25T18:19:00.347 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.345+0000 7fc1c0481640 10 monclient: init 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.346+0000 7fc1c0481640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.346+0000 7fc1c0481640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.346+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b80a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b8142520 con 0x7fc1b813f710 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b28a0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b2a30 con 0x7fc1b81430c0 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.348 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.347+0000 7fc1c0481640 10 monclient(hunting): authenticate will time out at 7849.608687s 2024-03-25T18:19:00.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.349+0000 7fc1be21d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc1b80a98f0 0x7fc1b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:57630/0 (socket says 172.21.15.46:57630) 2024-03-25T18:19:00.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.349+0000 7fc1be21d640 1 -- 172.21.15.46:0/2174697000 learned_addr learned my addr 172.21.15.46:0/2174697000 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:00.363 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608937716 0 0) 0x7fc1b80b28a0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.363 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.363 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.363 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:19:00.364 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.362+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a4003520 con 0x7fc1b80a98f0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013070627 0 0) 0x7fc1a4003520 con 0x7fc1b80a98f0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 >> v1:172.21.15.46:6790/0 conn(0x7fc1b81430c0 legacy=0x7fc1b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 >> v1:172.21.15.46:6789/0 conn(0x7fc1b813f710 legacy=0x7fc1b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 monclient: found mon.noname-c 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b80b44c0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: _finish_auth 0 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:00.372 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1b4001690 con 0x7fc1b80a98f0 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1bd21b640 10 monclient: dump: 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1c0481640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:19:00.373 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.371+0000 7fc1c0481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1b40017f0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 1 -- 172.21.15.46:0/2174697000 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1b4001c40 con 0x7fc1b80a98f0 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1bd21b640 10 monclient: dump: 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:00.379 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1c0481640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:00.380 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.378+0000 7fc1c0481640 4 set_mon_vals no callback set 2024-03-25T18:19:00.381 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 10 monclient: shutdown 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 1 -- 172.21.15.46:0/2174697000 >> v1:172.21.15.164:6789/0 conn(0x7fc1b80a98f0 legacy=0x7fc1b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 1 -- 172.21.15.46:0/2174697000 shutdown_connections 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 1 -- 172.21.15.46:0/2174697000 >> 172.21.15.46:0/2174697000 conn(0x7fc1b80b7ea0 msgr2=0x7fc1b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 1 -- 172.21.15.46:0/2174697000 shutdown_connections 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.379+0000 7fc1c0481640 1 -- 172.21.15.46:0/2174697000 wait complete. 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.380+0000 7fc1c0481640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.380+0000 7fc1c0481640 10 monclient: monmap: 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:00.382 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 librados: starting msgr at 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 librados: starting objecter 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 Processor -- start 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 -- start start 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 librados: setting wanted keys 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 librados: calling monclient init 2024-03-25T18:19:00.383 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient: init 2024-03-25T18:19:00.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:00.384 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b80a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b813f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b81430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b81db7d0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b1420 con 0x7fc1b813f710 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b15b0 con 0x7fc1b81430c0 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.385 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.381+0000 7fc1c0481640 10 monclient(hunting): authenticate will time out at 7849.643477s 2024-03-25T18:19:00.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.398+0000 7fc1bea1e640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7fc1b81430c0 0x7fc1b81d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:47448/0 (socket says 172.21.15.46:47448) 2024-03-25T18:19:00.399 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.398+0000 7fc1bea1e640 1 -- 172.21.15.46:0/2350971065 learned_addr learned my addr 172.21.15.46:0/2350971065 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:00.405 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.404+0000 7fc1aeffd640 10 client.?.objecter ms_handle_connect 0x7fc1b81430c0 2024-03-25T18:19:00.405 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.404+0000 7fc1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.406+0000 7fc1aeffd640 10 client.?.objecter ms_handle_connect 0x7fc1b80a98f0 2024-03-25T18:19:00.407 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.406+0000 7fc1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.409+0000 7fc1aeffd640 10 client.?.objecter ms_handle_connect 0x7fc1b813f710 2024-03-25T18:19:00.409 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.409+0000 7fc1aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931372849 0 0) 0x7fc1b81db7d0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:19:00.419 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.418+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c004eb0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.421+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632712948 0 0) 0x7fc1b80b1420 con 0x7fc1b813f710 2024-03-25T18:19:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.421+0000 7fc1aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.422 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.421+0000 7fc1aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.421+0000 7fc1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.421+0000 7fc1aeffd640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:19:00.423 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.422+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b81db7d0 con 0x7fc1b813f710 2024-03-25T18:19:00.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674510731 0 0) 0x7fc19c004eb0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b80b1420 con 0x7fc1b80a98f0 2024-03-25T18:19:00.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1b4001b00 con 0x7fc1b80a98f0 2024-03-25T18:19:00.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.429 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 10 monclient(hunting): dump: 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.430 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.428+0000 7fc1aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042161035 0 0) 0x7fc1b81db7d0 con 0x7fc1b813f710 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c004eb0 con 0x7fc1b813f710 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1a80018a0 con 0x7fc1b813f710 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient(hunting): dump: 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.436 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3222144905 0 0) 0x7fc1b80b1420 con 0x7fc1b80a98f0 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 >> v1:172.21.15.46:6790/0 conn(0x7fc1b81430c0 legacy=0x7fc1b81d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 >> v1:172.21.15.164:6789/0 conn(0x7fc1b813f710 legacy=0x7fc1b80b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 monclient: found mon.a 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:00.437 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b80b1740 con 0x7fc1b80a98f0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 10 monclient: _renew_subs 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b81db1a0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 10 monclient: _renew_subs 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b80b08f0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.434+0000 7fc1c0481640 1 librados: init done 2024-03-25T18:19:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.436+0000 7fc1c0481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.436+0000 7fc1c0481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.436+0000 7fc1c0481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:00.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.436+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b80627b0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.656+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1b4001d20 con 0x7fc1b80a98f0 2024-03-25T18:19:00.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.656+0000 7fc1aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:00.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.656+0000 7fc1bca1a640 4 set_mon_vals no callback set 2024-03-25T18:19:00.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.754+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fc1b4004d90 con 0x7fc1b80a98f0 2024-03-25T18:19:00.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.754+0000 7fc1aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.754+0000 7fc1aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:00.755 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.754+0000 7fc1aeffd640 10 monclient: dump: 2024-03-25T18:19:00.755 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:00.756 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.755+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fc1b401ad60 con 0x7fc1b80a98f0 2024-03-25T18:19:00.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.756+0000 7fc1bda1c640 10 monclient: get_auth_request con 0x7fc19c06de80 auth_method 0 2024-03-25T18:19:00.757 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.756+0000 7fc1aeffd640 10 client.4683.objecter ms_handle_connect 0x7fc19c06de80 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7fc1b4054990 con 0x7fc1b80a98f0 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 10 client.4683.objecter ms_dispatch 0x7fc1b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1b40504a0 con 0x7fc1b80a98f0 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.762 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.761+0000 7fc1aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:00.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.991+0000 7fc1c0481640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.991+0000 7fc1c0481640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.991+0000 7fc1c0481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:00.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:00.991+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1b80abae0 con 0x7fc1b80a98f0 2024-03-25T18:19:01.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.018+0000 7fc1aeffd640 1 -- 172.21.15.46:0/2350971065 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7fc1b40523d0 con 0x7fc1b80a98f0 2024-03-25T18:19:01.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.018+0000 7fc1aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.019 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.018+0000 7fc1aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:01.019 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:01.019 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:01.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 10 librados: watch_flush enter 2024-03-25T18:19:01.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 10 librados: watch_flush exit 2024-03-25T18:19:01.022 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 10 client.4683.objecter successfully canceled tick 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 >> v1:172.21.15.164:6800/1612084612 conn(0x7fc19c06de80 legacy=0x7fc19c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 10 monclient: shutdown 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 >> v1:172.21.15.46:6789/0 conn(0x7fc1b80a98f0 legacy=0x7fc1b81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 shutdown_connections 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 >> 172.21.15.46:0/2350971065 conn(0x7fc1b80b7ea0 msgr2=0x7fc1b813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 shutdown_connections 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 -- 172.21.15.46:0/2350971065 wait complete. 2024-03-25T18:19:01.023 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:01.021+0000 7fc1c0481640 1 librados: shutdown 2024-03-25T18:19:01.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:07.839 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:07.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.951+0000 7f91b413b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:07.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.951+0000 7f91b413b640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.952 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.951+0000 7f91b413b640 10 monclient: monmap: 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:07.952627+0000 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:07.952627+0000 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:07.953 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:07.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.952+0000 7f91b413b640 1 Processor -- start 2024-03-25T18:19:07.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.953+0000 7f91b413b640 1 -- start start 2024-03-25T18:19:07.954 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.953+0000 7f91b413b640 10 monclient: init 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.953+0000 7f91b413b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.953+0000 7f91b413b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.953+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac0b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac0b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac14be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0af940 con 0x7f91ac14be90 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0b0a70 con 0x7f91ac0b4d60 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0b0d50 con 0x7f91ac0b1730 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.955 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.954+0000 7f91b413b640 10 monclient(hunting): authenticate will time out at 7857.215783s 2024-03-25T18:19:07.959 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.959+0000 7f91b1ed7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f91ac0b4d60 0x7f91ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:37554/0 (socket says 172.21.15.46:37554) 2024-03-25T18:19:07.959 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.959+0000 7f91b1ed7640 1 -- 172.21.15.46:0/3608958307 learned_addr learned my addr 172.21.15.46:0/3608958307 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471118269 0 0) 0x7f91ac0af940 con 0x7f91ac14be90 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:19:07.971 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.970+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9198003520 con 0x7f91ac14be90 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571104376 0 0) 0x7f91ac0b0a70 con 0x7f91ac0b4d60 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:19:07.974 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.973+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac0af940 con 0x7f91ac0b4d60 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883513900 0 0) 0x7f91ac0af940 con 0x7f91ac0b4d60 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 >> v1:172.21.15.46:6790/0 conn(0x7f91ac0b1730 legacy=0x7f91ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 >> v1:172.21.15.46:6789/0 conn(0x7f91ac14be90 legacy=0x7f91ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: _finish_hunting 0 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 monclient: found mon.noname-c 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac0b1030 con 0x7f91ac0b4d60 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: _finish_auth 0 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: _check_auth_tickets 2024-03-25T18:19:07.979 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f91a8001670 con 0x7f91ac0b4d60 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b413b640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b413b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b0ed5640 10 monclient: dump: 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:07.980 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:07.981 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:07.981 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.978+0000 7f91b413b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.986+0000 7f91b0ed5640 1 -- 172.21.15.46:0/3608958307 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a8001850 con 0x7f91ac0b4d60 2024-03-25T18:19:07.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.986+0000 7f91b0ed5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:07.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.986+0000 7f91b413b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:07.987 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.986+0000 7f91b413b640 4 set_mon_vals no callback set 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 10 monclient: shutdown 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 1 -- 172.21.15.46:0/3608958307 >> v1:172.21.15.164:6789/0 conn(0x7f91ac0b4d60 legacy=0x7f91ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 1 -- 172.21.15.46:0/3608958307 shutdown_connections 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 1 -- 172.21.15.46:0/3608958307 >> 172.21.15.46:0/3608958307 conn(0x7f91ac0617a0 msgr2=0x7f91ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 1 -- 172.21.15.46:0/3608958307 shutdown_connections 2024-03-25T18:19:07.988 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.987+0000 7f91b413b640 1 -- 172.21.15.46:0/3608958307 wait complete. 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.988+0000 7f91b413b640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.988+0000 7f91b413b640 10 monclient: monmap: 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:07.989 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:07.990 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.988+0000 7f91b413b640 1 librados: starting msgr at 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.988+0000 7f91b413b640 1 librados: starting objecter 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.988+0000 7f91b413b640 1 Processor -- start 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 -- start start 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 librados: setting wanted keys 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 librados: calling monclient init 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient: init 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac0b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac0b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac14be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac142250 con 0x7f91ac14be90 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0a9ef0 con 0x7f91ac0b4d60 2024-03-25T18:19:07.991 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac142e80 con 0x7f91ac0b1730 2024-03-25T18:19:07.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.992 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.989+0000 7f91b413b640 10 monclient(hunting): authenticate will time out at 7857.251536s 2024-03-25T18:19:07.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.994+0000 7f91b16d6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f91ac0b4d60 0x7f91ac142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:37570/0 (socket says 172.21.15.46:37570) 2024-03-25T18:19:07.994 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.994+0000 7f91b16d6640 1 -- 172.21.15.46:0/2801036071 learned_addr learned my addr 172.21.15.46:0/2801036071 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:07.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.995+0000 7f91a27fc640 10 client.?.objecter ms_handle_connect 0x7f91ac0b4d60 2024-03-25T18:19:07.996 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:07.995+0000 7f91a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.000+0000 7f91a27fc640 10 client.?.objecter ms_handle_connect 0x7f91ac0b1730 2024-03-25T18:19:08.000 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.000+0000 7f91a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.006 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.005+0000 7f91a27fc640 10 client.?.objecter ms_handle_connect 0x7f91ac14be90 2024-03-25T18:19:08.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.006+0000 7f91a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655830473 0 0) 0x7f91ac0a9ef0 con 0x7f91ac0b4d60 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:19:08.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.008+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9190004eb0 con 0x7f91ac0b4d60 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231234431 0 0) 0x7f91ac142e80 con 0x7f91ac0b1730 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:19:08.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.011+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac0a9ef0 con 0x7f91ac0b1730 2024-03-25T18:19:08.026 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655503107 0 0) 0x7f9190004eb0 con 0x7f91ac0b4d60 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ac142e80 con 0x7f91ac0b4d60 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f919c001a30 con 0x7f91ac0b4d60 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:08.027 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.025+0000 7f91a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691559651 0 0) 0x7f91ac142250 con 0x7f91ac14be90 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.028 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:19:08.029 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.026+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9190004eb0 con 0x7f91ac14be90 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194914672 0 0) 0x7f91ac0a9ef0 con 0x7f91ac0b1730 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ac142250 con 0x7f91ac0b1730 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f91a8001630 con 0x7f91ac0b1730 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.030+0000 7f91a27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.031 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:08.032 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:08.032 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:08.032 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:08.032 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:08.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4091532015 0 0) 0x7f91ac142e80 con 0x7f91ac0b4d60 2024-03-25T18:19:08.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 >> v1:172.21.15.46:6790/0 conn(0x7f91ac0b1730 legacy=0x7f91ac141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.036 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 >> v1:172.21.15.46:6789/0 conn(0x7f91ac14be90 legacy=0x7f91ac1e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 1 monclient: found mon.b 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac140380 con 0x7f91ac0b4d60 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 10 monclient: _renew_subs 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac140630 con 0x7f91ac0b4d60 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:08.037 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 10 monclient: _renew_subs 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac13ae40 con 0x7f91ac0b4d60 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.035+0000 7f91b413b640 1 librados: init done 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.036+0000 7f91b413b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.036+0000 7f91b413b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.036+0000 7f91b413b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:08.038 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.036+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac142090 con 0x7f91ac0b4d60 2024-03-25T18:19:08.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f919c001c70 con 0x7f91ac0b4d60 2024-03-25T18:19:08.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:08.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f919c003a60 con 0x7f91ac0b4d60 2024-03-25T18:19:08.312 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a27fc640 10 monclient: dump: 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:08.313 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.311+0000 7f91a3fff640 4 set_mon_vals no callback set 2024-03-25T18:19:08.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.312+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f919c003c80 con 0x7f91ac0b4d60 2024-03-25T18:19:08.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.313+0000 7f91b1ed7640 10 monclient: get_auth_request con 0x7f919006dfb0 auth_method 0 2024-03-25T18:19:08.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.313+0000 7f91a27fc640 10 client.4690.objecter ms_handle_connect 0x7f919006dfb0 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f919c054820 con 0x7f91ac0b4d60 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 10 client.4690.objecter ms_dispatch 0x7f91ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T18:19:08.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f919c050330 con 0x7f91ac0b4d60 2024-03-25T18:19:08.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.318+0000 7f91a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:08.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.556+0000 7f91b413b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.556+0000 7f91b413b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.556+0000 7f91b413b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:08.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.556+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91ac0627b0 con 0x7f91ac0b4d60 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.613+0000 7f91a27fc640 1 -- 172.21.15.46:0/2801036071 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f919c052260 con 0x7f91ac0b4d60 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.613+0000 7f91a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.613+0000 7f91a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 10 librados: watch_flush enter 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 10 librados: watch_flush exit 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 10 client.4690.objecter successfully canceled tick 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 >> v1:172.21.15.164:6800/1612084612 conn(0x7f919006dfb0 legacy=0x7f9190070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.617 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 10 monclient: shutdown 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 >> v1:172.21.15.164:6789/0 conn(0x7f91ac0b4d60 legacy=0x7f91ac142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 shutdown_connections 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 >> 172.21.15.46:0/2801036071 conn(0x7f91ac0617a0 msgr2=0x7f91ac0b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.615+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 shutdown_connections 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.616+0000 7f91b413b640 1 -- 172.21.15.46:0/2801036071 wait complete. 2024-03-25T18:19:08.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:08.616+0000 7f91b413b640 1 librados: shutdown 2024-03-25T18:19:08.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:10.987 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:46.905 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:10.996 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:46.914 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.029 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:10:46.946 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.048 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:46.919 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.052 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:46.922 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.057 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:46.974 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.068 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:46.985 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.083 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:10:46.954 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:11.104 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:10:47.022 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.433 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.537+0000 7fa1a07a8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.537+0000 7fa1a07a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.537+0000 7fa1a07a8640 10 monclient: monmap: 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:15.538190+0000 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:15.538190+0000 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:15.539 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:15.539 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:15.539 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.538+0000 7fa1a07a8640 1 Processor -- start 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.538+0000 7fa1a07a8640 1 -- start start 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.538+0000 7fa1a07a8640 10 monclient: init 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa1980ae6e0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa19806f080 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa1980704e0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980718d0 con 0x7fa19806f080 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19806fa30 con 0x7fa1980704e0 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa198070090 con 0x7fa1980ae6e0 2024-03-25T18:19:15.540 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.541 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.539+0000 7fa1a07a8640 10 monclient(hunting): authenticate will time out at 7864.800844s 2024-03-25T18:19:15.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.553+0000 7fa19dd43640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa19806f080 0x7fa198071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52000/0 (socket says 172.21.15.46:52000) 2024-03-25T18:19:15.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.553+0000 7fa19dd43640 1 -- 172.21.15.46:0/143086017 learned_addr learned my addr 172.21.15.46:0/143086017 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:15.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.560+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517561089 0 0) 0x7fa19806fa30 con 0x7fa1980704e0 2024-03-25T18:19:15.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.560+0000 7fa19d542640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.561 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.560+0000 7fa19d542640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.560+0000 7fa19d542640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.560+0000 7fa19d542640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.561+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184003520 con 0x7fa1980704e0 2024-03-25T18:19:15.567 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544781687 0 0) 0x7fa1980718d0 con 0x7fa19806f080 2024-03-25T18:19:15.567 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:19:15.568 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.566+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19806fa30 con 0x7fa19806f080 2024-03-25T18:19:15.571 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482585707 0 0) 0x7fa198070090 con 0x7fa1980ae6e0 2024-03-25T18:19:15.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:19:15.572 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.571+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1980718d0 con 0x7fa1980ae6e0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547475763 0 0) 0x7fa184003520 con 0x7fa1980704e0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 >> v1:172.21.15.46:6790/0 conn(0x7fa1980ae6e0 legacy=0x7fa19813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 >> v1:172.21.15.46:6789/0 conn(0x7fa19806f080 legacy=0x7fa198071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 monclient: found mon.noname-c 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19806f510 con 0x7fa1980704e0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.574 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa190001a10 con 0x7fa1980704e0 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa19d542640 10 monclient: dump: 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa1a07a8640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:19:15.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.572+0000 7fa1a07a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa190001b70 con 0x7fa1980704e0 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 1 -- 172.21.15.46:0/143086017 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa190001f80 con 0x7fa1980704e0 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa1a07a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa1a07a8640 4 set_mon_vals no callback set 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.578+0000 7fa19d542640 10 monclient: dump: 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:15.580 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 10 monclient: shutdown 2024-03-25T18:19:15.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:15.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 1 -- 172.21.15.46:0/143086017 >> v1:172.21.15.164:6789/0 conn(0x7fa1980704e0 legacy=0x7fa19814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 1 -- 172.21.15.46:0/143086017 shutdown_connections 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 1 -- 172.21.15.46:0/143086017 >> 172.21.15.46:0/143086017 conn(0x7fa19806b630 msgr2=0x7fa1980b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 1 -- 172.21.15.46:0/143086017 shutdown_connections 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.579+0000 7fa1a07a8640 1 -- 172.21.15.46:0/143086017 wait complete. 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.580+0000 7fa1a07a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.580+0000 7fa1a07a8640 10 monclient: monmap: 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:15.582 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.580+0000 7fa1a07a8640 1 librados: starting msgr at 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.580+0000 7fa1a07a8640 1 librados: starting objecter 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 Processor -- start 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 -- start start 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 librados: setting wanted keys 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 librados: calling monclient init 2024-03-25T18:19:15.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient: init 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa1980704e0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa1980ae6e0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa198142bf0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980a9470 con 0x7fa198142bf0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980aaf10 con 0x7fa1980704e0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1981d73f0 con 0x7fa1980ae6e0 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.581+0000 7fa1a07a8640 10 monclient(hunting): authenticate will time out at 7864.843477s 2024-03-25T18:19:15.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.591+0000 7fa19ed45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa198142bf0 0x7fa198138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52028/0 (socket says 172.21.15.46:52028) 2024-03-25T18:19:15.592 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.591+0000 7fa19ed45640 1 -- 172.21.15.46:0/903737968 learned_addr learned my addr 172.21.15.46:0/903737968 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:15.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.597+0000 7fa18effd640 10 client.?.objecter ms_handle_connect 0x7fa198142bf0 2024-03-25T18:19:15.598 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.597+0000 7fa18effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.606+0000 7fa18effd640 10 client.?.objecter ms_handle_connect 0x7fa1980704e0 2024-03-25T18:19:15.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.606+0000 7fa18effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.606+0000 7fa18effd640 10 client.?.objecter ms_handle_connect 0x7fa1980ae6e0 2024-03-25T18:19:15.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.606+0000 7fa18effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625402603 0 0) 0x7fa1980a9470 con 0x7fa198142bf0 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:19:15.618 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.617+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c004eb0 con 0x7fa198142bf0 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235547753 0 0) 0x7fa1981d73f0 con 0x7fa1980ae6e0 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:19:15.627 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.626+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1980a9470 con 0x7fa1980ae6e0 2024-03-25T18:19:15.630 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994346282 0 0) 0x7fa17c004eb0 con 0x7fa198142bf0 2024-03-25T18:19:15.630 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1981d73f0 con 0x7fa198142bf0 2024-03-25T18:19:15.630 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa190001ec0 con 0x7fa198142bf0 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 10 monclient(hunting): dump: 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.631 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.629+0000 7fa18effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:15.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.633+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610654073 0 0) 0x7fa1980aaf10 con 0x7fa1980704e0 2024-03-25T18:19:15.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.634+0000 7fa18effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.634 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.634+0000 7fa18effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.634+0000 7fa18effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.634+0000 7fa18effd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:19:15.635 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.634+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c004eb0 con 0x7fa1980704e0 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203024613 0 0) 0x7fa1981d73f0 con 0x7fa198142bf0 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 >> v1:172.21.15.46:6790/0 conn(0x7fa1980ae6e0 legacy=0x7fa1980a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 >> v1:172.21.15.164:6789/0 conn(0x7fa1980704e0 legacy=0x7fa1981d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 1 monclient: found mon.a 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1981d7580 con 0x7fa198142bf0 2024-03-25T18:19:15.640 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa18effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 10 monclient: _renew_subs 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1980aa010 con 0x7fa198142bf0 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 10 monclient: _renew_subs 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198139240 con 0x7fa198142bf0 2024-03-25T18:19:15.641 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.639+0000 7fa1a07a8640 1 librados: init done 2024-03-25T18:19:15.642 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.640+0000 7fa1a07a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.642 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.640+0000 7fa1a07a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.642 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.640+0000 7fa1a07a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:15.642 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.640+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1980a9620 con 0x7fa198142bf0 2024-03-25T18:19:15.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.974+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa190003c20 con 0x7fa198142bf0 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fa1900015d0 con 0x7fa198142bf0 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 10 monclient: dump: 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:15.976 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:15.977 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:15.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa19cd41640 4 set_mon_vals no callback set 2024-03-25T18:19:15.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.975+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fa190003f50 con 0x7fa198142bf0 2024-03-25T18:19:15.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.976+0000 7fa19e544640 10 monclient: get_auth_request con 0x7fa17c06de20 auth_method 0 2024-03-25T18:19:15.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.976+0000 7fa18effd640 10 client.4695.objecter ms_handle_connect 0x7fa17c06de20 2024-03-25T18:19:15.982 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7fa190054a30 con 0x7fa198142bf0 2024-03-25T18:19:15.982 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 10 client.4695.objecter ms_dispatch 0x7fa198000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa190050540 con 0x7fa198142bf0 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.983 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:15.982+0000 7fa18effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:16.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.242+0000 7fa1a07a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.242+0000 7fa1a07a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.242+0000 7fa1a07a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:16.243 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.242+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1980ad6c0 con 0x7fa198142bf0 2024-03-25T18:19:16.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.285+0000 7fa18effd640 1 -- 172.21.15.46:0/903737968 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7fa190052470 con 0x7fa198142bf0 2024-03-25T18:19:16.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.285+0000 7fa18effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.286 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.285+0000 7fa18effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:16.286 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:16.287 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.287+0000 7fa1a07a8640 10 librados: watch_flush enter 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 10 librados: watch_flush exit 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 >> v1:172.21.15.164:6800/1612084612 conn(0x7fa17c06de20 legacy=0x7fa17c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 10 monclient: shutdown 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 >> v1:172.21.15.46:6789/0 conn(0x7fa198142bf0 legacy=0x7fa198138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.289 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 shutdown_connections 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 >> 172.21.15.46:0/903737968 conn(0x7fa19806b630 msgr2=0x7fa1980b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 shutdown_connections 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 -- 172.21.15.46:0/903737968 wait complete. 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:16.288+0000 7fa1a07a8640 1 librados: shutdown 2024-03-25T18:19:16.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:23.107 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:23.222 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.222+0000 7f221aec0640 10 monclient: get_monmap_and_config 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.222+0000 7f221aec0640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.222+0000 7f221aec0640 10 monclient: monmap: 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:23.222906+0000 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:23.222906+0000 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:23.223 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.224 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 1 Processor -- start 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 1 -- start start 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient: init 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-b con 0x7f2214061890 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-a con 0x7f221406e740 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-c con 0x7f22140ae280 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140ae830 con 0x7f221406e740 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221406f720 con 0x7f22140ae280 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140ada80 con 0x7f2214061890 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.225 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.223+0000 7f221aec0640 10 monclient(hunting): authenticate will time out at 7872.485527s 2024-03-25T18:19:23.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.236+0000 7f2213fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f221406e740 0x7f2214145820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56758/0 (socket says 172.21.15.46:56758) 2024-03-25T18:19:23.237 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.236+0000 7f2213fff640 1 -- 172.21.15.46:0/1161982458 learned_addr learned my addr 172.21.15.46:0/1161982458 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:23.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125413891 0 0) 0x7f22140ae830 con 0x7f221406e740 2024-03-25T18:19:23.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:19:23.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.249+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204003520 con 0x7f221406e740 2024-03-25T18:19:23.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057817121 0 0) 0x7f22140ada80 con 0x7f2214061890 2024-03-25T18:19:23.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.266 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:19:23.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.266+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22140ae830 con 0x7f2214061890 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197528410 0 0) 0x7f221406f720 con 0x7f22140ae280 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:19:23.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.277+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22140ada80 con 0x7f22140ae280 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244375222 0 0) 0x7f2204003520 con 0x7f221406e740 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 >> v1:172.21.15.46:6790/0 conn(0x7f2214061890 legacy=0x7f221413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 >> v1:172.21.15.164:6789/0 conn(0x7f22140ae280 legacy=0x7f2214145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 monclient: found mon.noname-a 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221413fbd0 con 0x7f221406e740 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:23.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2200001690 con 0x7f221406e740 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f221aec0640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f221aec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f22137fe640 10 monclient: dump: 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:23.281 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.282 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.278+0000 7f221aec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22000017f0 con 0x7f221406e740 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 1 -- 172.21.15.46:0/1161982458 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2200001c00 con 0x7f221406e740 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f221aec0640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f221aec0640 4 set_mon_vals no callback set 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.289+0000 7f22137fe640 10 monclient: dump: 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 10 monclient: shutdown 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 1 -- 172.21.15.46:0/1161982458 >> v1:172.21.15.46:6789/0 conn(0x7f221406e740 legacy=0x7f2214145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 1 -- 172.21.15.46:0/1161982458 shutdown_connections 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 1 -- 172.21.15.46:0/1161982458 >> 172.21.15.46:0/1161982458 conn(0x7f22140b0ef0 msgr2=0x7f22140b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 1 -- 172.21.15.46:0/1161982458 shutdown_connections 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.290+0000 7f221aec0640 1 -- 172.21.15.46:0/1161982458 wait complete. 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 10 monclient: monmap: 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:23.294 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 librados: starting msgr at 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 librados: starting objecter 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 Processor -- start 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 -- start start 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 librados: setting wanted keys 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 1 librados: calling monclient init 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.291+0000 7f221aec0640 10 monclient: init 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-c con 0x7f2214061890 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-a con 0x7f22140ae280 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient(hunting): picked mon.noname-b con 0x7f2214146970 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221413d830 con 0x7f22140ae280 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221413db10 con 0x7f2214146970 2024-03-25T18:19:23.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221413c200 con 0x7f2214061890 2024-03-25T18:19:23.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.292+0000 7f221aec0640 10 monclient(hunting): authenticate will time out at 7872.553992s 2024-03-25T18:19:23.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.300+0000 7f2218c5c640 1 --1- >> v1:172.21.15.46:6790/0 conn(0x7f2214061890 0x7f221413ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6790/0 says I am v1:172.21.15.46:49582/0 (socket says 172.21.15.46:49582) 2024-03-25T18:19:23.301 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.300+0000 7f2218c5c640 1 -- 172.21.15.46:0/2112248077 learned_addr learned my addr 172.21.15.46:0/2112248077 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:23.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.302+0000 7f22117fa640 10 client.?.objecter ms_handle_connect 0x7f2214061890 2024-03-25T18:19:23.303 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.302+0000 7f22117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.306+0000 7f22117fa640 10 client.?.objecter ms_handle_connect 0x7f22140ae280 2024-03-25T18:19:23.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.306+0000 7f22117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.308+0000 7f22117fa640 10 client.?.objecter ms_handle_connect 0x7f2214146970 2024-03-25T18:19:23.308 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.308+0000 7f22117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994917569 0 0) 0x7f221413d830 con 0x7f22140ae280 2024-03-25T18:19:23.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:19:23.315 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.313+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8004eb0 con 0x7f22140ae280 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390191771 0 0) 0x7f221413c200 con 0x7f2214061890 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.320+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221413d830 con 0x7f2214061890 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814259315 0 0) 0x7f21f8004eb0 con 0x7f22140ae280 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f221413c200 con 0x7f22140ae280 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2200001b00 con 0x7f22140ae280 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 10 monclient(hunting): dump: 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:23.331 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:23.332 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:23.332 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.332 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.330+0000 7f22117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224692576 0 0) 0x7f221413db10 con 0x7f2214146970 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:19:23.337 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.336+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8004eb0 con 0x7f2214146970 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3420847819 0 0) 0x7f221413d830 con 0x7f2214061890 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f221413db10 con 0x7f2214061890 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f22080019b0 con 0x7f2214061890 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.349+0000 7f22117fa640 10 monclient(hunting): dump: 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.350 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.351 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.351 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:23.351 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:23.351 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:23.351 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2055628972 0 0) 0x7f221413c200 con 0x7f22140ae280 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 >> v1:172.21.15.46:6790/0 conn(0x7f2214061890 legacy=0x7f221413ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 >> v1:172.21.15.164:6789/0 conn(0x7f2214146970 legacy=0x7f22141e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 1 monclient: found mon.a 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221413abd0 con 0x7f22140ae280 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f22117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:23.355 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 10 monclient: _renew_subs 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221413e280 con 0x7f22140ae280 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 10 monclient: _renew_subs 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141e3950 con 0x7f22140ae280 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.353+0000 7f221aec0640 1 librados: init done 2024-03-25T18:19:23.356 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.354+0000 7f221aec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.354+0000 7f221aec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.354+0000 7f221aec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.357 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.354+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22140627b0 con 0x7f22140ae280 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2200001220 con 0x7f22140ae280 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2200004c30 con 0x7f22140ae280 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:23.695 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f22117fa640 10 monclient: dump: 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:23.696 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.694+0000 7f2212ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:23.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.695+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f220001ac00 con 0x7f22140ae280 2024-03-25T18:19:23.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.696+0000 7f2218c5c640 10 monclient: get_auth_request con 0x7f21f806dfb0 auth_method 0 2024-03-25T18:19:23.697 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.696+0000 7f22117fa640 10 client.4701.objecter ms_handle_connect 0x7f21f806dfb0 2024-03-25T18:19:23.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f2200054890 con 0x7f22140ae280 2024-03-25T18:19:23.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 10 client.4701.objecter ms_dispatch 0x7f2214000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:23.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:23.702 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:23.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T18:19:23.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22000501d0 con 0x7f22140ae280 2024-03-25T18:19:23.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.703 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.701+0000 7f22117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:23.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.942+0000 7f221aec0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.942+0000 7f221aec0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.942+0000 7f221aec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:23.943 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:23.942+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2214069f40 con 0x7f22140ae280 2024-03-25T18:19:24.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.046+0000 7f22117fa640 1 -- 172.21.15.46:0/2112248077 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f22000522d0 con 0x7f22140ae280 2024-03-25T18:19:24.047 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.046+0000 7f22117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.046+0000 7f22117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.049+0000 7f221aec0640 10 librados: watch_flush enter 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.049+0000 7f221aec0640 10 librados: watch_flush exit 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.049+0000 7f221aec0640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.049+0000 7f221aec0640 10 client.4701.objecter successfully canceled tick 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.049+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 >> v1:172.21.15.164:6800/1612084612 conn(0x7f21f806dfb0 legacy=0x7f21f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 10 monclient: shutdown 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:24.051 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 >> v1:172.21.15.46:6789/0 conn(0x7f22140ae280 legacy=0x7f22141394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 shutdown_connections 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 >> 172.21.15.46:0/2112248077 conn(0x7f22140b0ef0 msgr2=0x7f22140aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 shutdown_connections 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 -- 172.21.15.46:0/2112248077 wait complete. 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:24.050+0000 7f221aec0640 1 librados: shutdown 2024-03-25T18:19:24.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:30.868 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:30.959 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:06.877 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.974+0000 7f64c633d640 10 monclient: get_monmap_and_config 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.974+0000 7f64c633d640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.974+0000 7f64c633d640 10 monclient: monmap: 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:30.975186+0000 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:30.975186+0000 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.975 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:30.976 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:30.976 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:30.976 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:30.976 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 1 Processor -- start 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 1 -- start start 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient: init 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-c con 0x7f64c00b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-a con 0x7f64c014be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00af940 con 0x7f64c014be90 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00b0a70 con 0x7f64c00b4d60 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00b0d50 con 0x7f64c00b1730 2024-03-25T18:19:30.977 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.978 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.975+0000 7f64c633d640 10 monclient(hunting): authenticate will time out at 7880.237417s 2024-03-25T18:19:30.978 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.976+0000 7f64c48da640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f64c014be90 0x7f64c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42246/0 (socket says 172.21.15.46:42246) 2024-03-25T18:19:30.978 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.976+0000 7f64bffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f64c00b4d60 0x7f64c013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50008/0 (socket says 172.21.15.46:50008) 2024-03-25T18:19:30.978 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:30.976+0000 7f64c48da640 1 -- 172.21.15.46:0/2185454749 learned_addr learned my addr 172.21.15.46:0/2185454749 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:31.010 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:06.928 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.035 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:06.953 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.065 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:06.982 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698590662 0 0) 0x7f64c00af940 con 0x7f64c014be90 2024-03-25T18:19:31.065 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:19:31.066 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.064+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8003520 con 0x7f64c014be90 2024-03-25T18:19:31.074 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257319017 0 0) 0x7f64c00b0d50 con 0x7f64c00b1730 2024-03-25T18:19:31.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:19:31.075 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.074+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c00af940 con 0x7f64c00b1730 2024-03-25T18:19:31.075 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:06.992 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748473859 0 0) 0x7f64c00b0a70 con 0x7f64c00b4d60 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:19:31.081 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.080+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c00b0d50 con 0x7f64c00b4d60 2024-03-25T18:19:31.082 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:06.952 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.091 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:06.962 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.092 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1605457919 0 0) 0x7f64a8003520 con 0x7f64c014be90 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 >> v1:172.21.15.46:6790/0 conn(0x7f64c00b1730 legacy=0x7f64c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 >> v1:172.21.15.164:6789/0 conn(0x7f64c00b4d60 legacy=0x7f64c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 monclient: found mon.noname-a 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c00b1030 con 0x7f64c014be90 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f64b8001a10 con 0x7f64c014be90 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64c633d640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64c633d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:31.093 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64beffd640 10 monclient: dump: 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.094 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.092+0000 7f64c633d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:31.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.0 v1:172.21.15.46:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b8001b70 con 0x7f64c014be90 2024-03-25T18:19:31.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:31.115 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 1 -- 172.21.15.46:0/2185454749 <== mon.0 v1:172.21.15.46:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f64b8001f80 con 0x7f64c014be90 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64beffd640 10 monclient: dump: 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:31.116 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 4 set_mon_vals no callback set 2024-03-25T18:19:31.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 10 monclient: shutdown 2024-03-25T18:19:31.118 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 1 -- 172.21.15.46:0/2185454749 >> v1:172.21.15.46:6789/0 conn(0x7f64c014be90 legacy=0x7f64c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 1 -- 172.21.15.46:0/2185454749 shutdown_connections 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 1 -- 172.21.15.46:0/2185454749 >> 172.21.15.46:0/2185454749 conn(0x7f64c00617a0 msgr2=0x7f64c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 1 -- 172.21.15.46:0/2185454749 shutdown_connections 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.115+0000 7f64c633d640 1 -- 172.21.15.46:0/2185454749 wait complete. 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.116+0000 7f64c633d640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.116+0000 7f64c633d640 10 monclient: monmap: 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:31.119 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.116+0000 7f64c633d640 1 librados: starting msgr at 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.116+0000 7f64c633d640 1 librados: starting objecter 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 Processor -- start 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 -- start start 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 librados: setting wanted keys 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 librados: calling monclient init 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient: init 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-a con 0x7f64c00b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 10 monclient(hunting): picked mon.noname-c con 0x7f64c014be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0134a00 con 0x7f64c00b1730 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0139050 con 0x7f64c00b4d60 2024-03-25T18:19:31.120 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.117+0000 7f64c633d640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c013a330 con 0x7f64c014be90 2024-03-25T18:19:31.121 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.118+0000 7f64c633d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.121 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.118+0000 7f64c633d640 10 monclient(hunting): authenticate will time out at 7880.379550s 2024-03-25T18:19:31.121 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:07.038 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.123 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:06.994 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.133+0000 7f64bf7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f64c00b4d60 0x7f64c0133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:50032/0 (socket says 172.21.15.46:50032) 2024-03-25T18:19:31.133 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.133+0000 7f64bf7fe640 1 -- 172.21.15.46:0/3208328772 learned_addr learned my addr 172.21.15.46:0/3208328772 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:31.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.134+0000 7f64bcff9640 10 client.?.objecter ms_handle_connect 0x7f64c00b4d60 2024-03-25T18:19:31.134 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.134+0000 7f64bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.157 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.157+0000 7f64bcff9640 10 client.?.objecter ms_handle_connect 0x7f64c014be90 2024-03-25T18:19:31.158 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.157+0000 7f64bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.183+0000 7f64bcff9640 10 client.?.objecter ms_handle_connect 0x7f64c00b1730 2024-03-25T18:19:31.184 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.183+0000 7f64bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517864753 0 0) 0x7f64c013a330 con 0x7f64c014be90 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:19:31.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.193+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac004eb0 con 0x7f64c014be90 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341687481 0 0) 0x7f64c0139050 con 0x7f64c00b4d60 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:19:31.202 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.201+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c013a330 con 0x7f64c00b4d60 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829618097 0 0) 0x7f64c0134a00 con 0x7f64c00b1730 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:19:31.207 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.206+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0139050 con 0x7f64c00b1730 2024-03-25T18:19:31.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909854271 0 0) 0x7f64ac004eb0 con 0x7f64c014be90 2024-03-25T18:19:31.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c0134a00 con 0x7f64c014be90 2024-03-25T18:19:31.217 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f64b8001e80 con 0x7f64c014be90 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 10 monclient(hunting): dump: 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.218 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.216+0000 7f64bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654227050 0 0) 0x7f64c0139050 con 0x7f64c00b1730 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64ac004eb0 con 0x7f64c00b1730 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f64b0001830 con 0x7f64c00b1730 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.222+0000 7f64bcff9640 10 monclient(hunting): dump: 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.223 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:31.224 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.256 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 727983677 0 0) 0x7f64ac004eb0 con 0x7f64c00b1730 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 >> v1:172.21.15.46:6790/0 conn(0x7f64c014be90 legacy=0x7f64c01342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 >> v1:172.21.15.164:6789/0 conn(0x7f64c00b4d60 legacy=0x7f64c0133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 1 monclient: found mon.a 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c013a610 con 0x7f64c00b1730 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:19:31.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 10 monclient: _renew_subs 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c013a8f0 con 0x7f64c00b1730 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 10 monclient: _renew_subs 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c0139730 con 0x7f64c00b1730 2024-03-25T18:19:31.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.255+0000 7f64c633d640 1 librados: init done 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.256+0000 7f64c633d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.256+0000 7f64c633d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.256+0000 7f64c633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.256+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c01391e0 con 0x7f64c00b1730 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.265+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b0001d40 con 0x7f64c00b1730 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.265+0000 7f64bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:31.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.265+0000 7f64be7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:31.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.575+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f64b0004820 con 0x7f64c00b1730 2024-03-25T18:19:31.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.575+0000 7f64bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.575+0000 7f64bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.575+0000 7f64bcff9640 10 monclient: dump: 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:31.577 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:31.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.576+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f64b001a870 con 0x7f64c00b1730 2024-03-25T18:19:31.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.577+0000 7f64bf7fe640 10 monclient: get_auth_request con 0x7f64ac06dfb0 auth_method 0 2024-03-25T18:19:31.578 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.577+0000 7f64bcff9640 10 client.4707.objecter ms_handle_connect 0x7f64ac06dfb0 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f64b00545f0 con 0x7f64c00b1730 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 10 client.4707.objecter ms_dispatch 0x7f64c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64b004ff30 con 0x7f64c00b1730 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.584 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.583+0000 7f64bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:31.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.841+0000 7f64c633d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.841+0000 7f64c633d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.841+0000 7f64c633d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:19:31.842 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.841+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64c00627b0 con 0x7f64c00b1730 2024-03-25T18:19:31.884 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.884+0000 7f64bcff9640 1 -- 172.21.15.46:0/3208328772 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f64b0052030 con 0x7f64c00b1730 2024-03-25T18:19:31.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.884+0000 7f64bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.885 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.884+0000 7f64bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:31.885 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:31.885 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:31.887 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.886+0000 7f64c633d640 10 librados: watch_flush enter 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.886+0000 7f64c633d640 10 librados: watch_flush exit 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.886+0000 7f64c633d640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.886+0000 7f64c633d640 10 client.4707.objecter successfully canceled tick 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.886+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 >> v1:172.21.15.164:6800/1612084612 conn(0x7f64ac06dfb0 legacy=0x7f64ac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 10 monclient: shutdown 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 >> v1:172.21.15.46:6789/0 conn(0x7f64c00b1730 legacy=0x7f64c013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 shutdown_connections 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 >> 172.21.15.46:0/3208328772 conn(0x7f64c00617a0 msgr2=0x7f64c00b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 shutdown_connections 2024-03-25T18:19:31.888 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 -- 172.21.15.46:0/3208328772 wait complete. 2024-03-25T18:19:31.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:31.887+0000 7f64c633d640 1 librados: shutdown 2024-03-25T18:19:31.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:38.707 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:38.825 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.823+0000 7f28b3860640 10 monclient: get_monmap_and_config 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.823+0000 7f28b3860640 10 monclient: build_initial_monmap 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.823+0000 7f28b3860640 10 monclient: monmap: 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:38.825140+0000 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:38.825140+0000 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:38.826 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.824+0000 7f28b3860640 1 Processor -- start 2024-03-25T18:19:38.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.824+0000 7f28b3860640 1 -- start start 2024-03-25T18:19:38.827 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.824+0000 7f28b3860640 10 monclient: init 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.824+0000 7f28b3860640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-b con 0x7f28ac0a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-c con 0x7f28ac13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-a con 0x7f28ac1430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac142520 con 0x7f28ac1430c0 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac0b28a0 con 0x7f28ac13f710 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac0b2a30 con 0x7f28ac0a98f0 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:38.828 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.825+0000 7f28b3860640 10 monclient(hunting): authenticate will time out at 7888.087767s 2024-03-25T18:19:38.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.832+0000 7f28b0dfb640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f28ac13f710 0x7f28ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47718/0 (socket says 172.21.15.46:47718) 2024-03-25T18:19:38.834 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.832+0000 7f28b0dfb640 1 -- 172.21.15.46:0/3382823849 learned_addr learned my addr 172.21.15.46:0/3382823849 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694361574 0 0) 0x7f28ac0b28a0 con 0x7f28ac13f710 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:19:38.839 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.836+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2898003520 con 0x7f28ac13f710 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.843+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4056274176 0 0) 0x7f2898003520 con 0x7f28ac13f710 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 >> v1:172.21.15.46:6790/0 conn(0x7f28ac0a98f0 legacy=0x7f28ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 >> v1:172.21.15.46:6789/0 conn(0x7f28ac1430c0 legacy=0x7f28ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 1 monclient: found mon.noname-c 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac0b44c0 con 0x7f28ac13f710 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:38.846 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f289c001690 con 0x7f28ac13f710 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28a3fff640 10 monclient: dump: 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.847 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28b3860640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:19:38.848 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.844+0000 7f28b3860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:38.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f289c0017f0 con 0x7f28ac13f710 2024-03-25T18:19:38.853 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:38.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 1 -- 172.21.15.46:0/3382823849 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f289c001c00 con 0x7f28ac13f710 2024-03-25T18:19:38.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28b3860640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:38.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28b3860640 4 set_mon_vals no callback set 2024-03-25T18:19:38.854 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.852+0000 7f28a3fff640 10 monclient: dump: 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.855 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 10 monclient: shutdown 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 1 -- 172.21.15.46:0/3382823849 >> v1:172.21.15.164:6789/0 conn(0x7f28ac13f710 legacy=0x7f28ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 1 -- 172.21.15.46:0/3382823849 shutdown_connections 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 1 -- 172.21.15.46:0/3382823849 >> 172.21.15.46:0/3382823849 conn(0x7f28ac0b7ea0 msgr2=0x7f28ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 1 -- 172.21.15.46:0/3382823849 shutdown_connections 2024-03-25T18:19:38.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.853+0000 7f28b3860640 1 -- 172.21.15.46:0/3382823849 wait complete. 2024-03-25T18:19:38.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 10 monclient: build_initial_monmap 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 10 monclient: monmap: 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 librados: starting msgr at 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 librados: starting objecter 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 Processor -- start 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 -- start start 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 librados: setting wanted keys 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 1 librados: calling monclient init 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.854+0000 7f28b3860640 10 monclient: init 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-a con 0x7f28ac0a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-b con 0x7f28ac13f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient(hunting): picked mon.noname-c con 0x7f28ac1430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac13be70 con 0x7f28ac0a98f0 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac131660 con 0x7f28ac13f710 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac1317f0 con 0x7f28ac1430c0 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:38.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.855+0000 7f28b3860640 10 monclient(hunting): authenticate will time out at 7888.118004s 2024-03-25T18:19:38.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.857+0000 7f28b0dfb640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f28ac13f710 0x7f28ac1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:47744/0 (socket says 172.21.15.46:47744) 2024-03-25T18:19:38.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.857+0000 7f28b0dfb640 1 -- 172.21.15.46:0/2771379690 learned_addr learned my addr 172.21.15.46:0/2771379690 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:38.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.863+0000 7f28a1ffb640 10 client.?.objecter ms_handle_connect 0x7f28ac1430c0 2024-03-25T18:19:38.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.863+0000 7f28a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.863+0000 7f28a1ffb640 10 client.?.objecter ms_handle_connect 0x7f28ac13f710 2024-03-25T18:19:38.865 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.863+0000 7f28a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496823245 0 0) 0x7f28ac131660 con 0x7f28ac13f710 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890004eb0 con 0x7f28ac13f710 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 client.?.objecter ms_handle_connect 0x7f28ac0a98f0 2024-03-25T18:19:38.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.874+0000 7f28a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71074196 0 0) 0x7f28ac1317f0 con 0x7f28ac1430c0 2024-03-25T18:19:38.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:19:38.883 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.880+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ac131660 con 0x7f28ac1430c0 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188325521 0 0) 0x7f28ac13be70 con 0x7f28ac0a98f0 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:19:38.889 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.887+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ac1317f0 con 0x7f28ac0a98f0 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978473550 0 0) 0x7f2890004eb0 con 0x7f28ac13f710 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28ac13be70 con 0x7f28ac13f710 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f289c001bc0 con 0x7f28ac13f710 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.895 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.893+0000 7f28a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707968709 0 0) 0x7f28ac131660 con 0x7f28ac1430c0 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2890004eb0 con 0x7f28ac1430c0 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f28a4001b10 con 0x7f28ac1430c0 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.903+0000 7f28a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.905 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:38.906 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043369691 0 0) 0x7f28ac1317f0 con 0x7f28ac0a98f0 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28ac131660 con 0x7f28ac0a98f0 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f28a8001830 con 0x7f28ac0a98f0 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.908 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2442224860 0 0) 0x7f28ac13be70 con 0x7f28ac13f710 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 >> v1:172.21.15.46:6790/0 conn(0x7f28ac1430c0 legacy=0x7f28ac1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 >> v1:172.21.15.46:6789/0 conn(0x7f28ac0a98f0 legacy=0x7f28ac130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 monclient: found mon.b 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:38.909 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac131980 con 0x7f28ac13f710 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 10 monclient: _renew_subs 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac135c60 con 0x7f28ac13f710 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 10 monclient: _renew_subs 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac0b7770 con 0x7f28ac13f710 2024-03-25T18:19:38.910 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.904+0000 7f28b3860640 1 librados: init done 2024-03-25T18:19:38.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.905+0000 7f28b3860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.906+0000 7f28b3860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.906+0000 7f28b3860640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:38.911 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:38.906+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ac074cc0 con 0x7f28ac13f710 2024-03-25T18:19:39.194 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.192+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f289c004a90 con 0x7f28ac13f710 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.192+0000 7f28a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.192+0000 7f28a37fe640 4 set_mon_vals no callback set 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.193+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f289c004da0 con 0x7f28ac13f710 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.193+0000 7f28a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.193+0000 7f28a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.193+0000 7f28a1ffb640 10 monclient: dump: 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.195 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.193+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f289c01ad70 con 0x7f28ac13f710 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.194+0000 7f28b15fc640 10 monclient: get_auth_request con 0x7f289006e180 auth_method 0 2024-03-25T18:19:39.196 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.194+0000 7f28a1ffb640 10 client.4714.objecter ms_handle_connect 0x7f289006e180 2024-03-25T18:19:39.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.201+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f289c0548e0 con 0x7f28ac13f710 2024-03-25T18:19:39.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.201+0000 7f28a1ffb640 10 client.4714.objecter ms_dispatch 0x7f28ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:39.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.201+0000 7f28a1ffb640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:39.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.201+0000 7f28a1ffb640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:39.203 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.201+0000 7f28a1ffb640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:19:39.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.202+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f289c0503f0 con 0x7f28ac13f710 2024-03-25T18:19:39.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.202+0000 7f28a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.204 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.202+0000 7f28a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:39.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.452+0000 7f28b3860640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.452+0000 7f28b3860640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.452+0000 7f28b3860640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:39.454 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.452+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28ac1371d0 con 0x7f28ac13f710 2024-03-25T18:19:39.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.475+0000 7f28a1ffb640 1 -- 172.21.15.46:0/2771379690 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f289c052320 con 0x7f28ac13f710 2024-03-25T18:19:39.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.475+0000 7f28a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.477 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.475+0000 7f28a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:39.477 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:39.477 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.477+0000 7f28b3860640 10 librados: watch_flush enter 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.477+0000 7f28b3860640 10 librados: watch_flush exit 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.477+0000 7f28b3860640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.477+0000 7f28b3860640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 >> v1:172.21.15.164:6800/1612084612 conn(0x7f289006e180 legacy=0x7f2890070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 10 monclient: shutdown 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.480 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 >> v1:172.21.15.164:6789/0 conn(0x7f28ac13f710 legacy=0x7f28ac1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 shutdown_connections 2024-03-25T18:19:39.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 >> 172.21.15.46:0/2771379690 conn(0x7f28ac0b7ea0 msgr2=0x7f28ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 shutdown_connections 2024-03-25T18:19:39.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 -- 172.21.15.46:0/2771379690 wait complete. 2024-03-25T18:19:39.481 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:39.478+0000 7f28b3860640 1 librados: shutdown 2024-03-25T18:19:39.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:46.299 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:46.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.393+0000 7f1c9ba31640 10 monclient: get_monmap_and_config 2024-03-25T18:19:46.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.393+0000 7f1c9ba31640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.395 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.393+0000 7f1c9ba31640 10 monclient: monmap: 2024-03-25T18:19:46.395 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:46.395 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:46.395401+0000 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:46.395401+0000 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:46.396 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.397 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.394+0000 7f1c9ba31640 1 Processor -- start 2024-03-25T18:19:46.397 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.394+0000 7f1c9ba31640 1 -- start start 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.394+0000 7f1c9ba31640 10 monclient: init 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-a con 0x7f1c940a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-b con 0x7f1c9413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-c con 0x7f1c941430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c94142520 con 0x7f1c940a98f0 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940b28a0 con 0x7f1c941430c0 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940b2a30 con 0x7f1c9413f710 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c9ba31640 10 monclient(hunting): authenticate will time out at 7895.657678s 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c997cd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c940a98f0 0x7f1c940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57058/0 (socket says 172.21.15.46:57058) 2024-03-25T18:19:46.398 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.395+0000 7f1c997cd640 1 -- 172.21.15.46:0/517687136 learned_addr learned my addr 172.21.15.46:0/517687136 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220945342 0 0) 0x7f1c94142520 con 0x7f1c940a98f0 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:19:46.412 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.410+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c80003520 con 0x7f1c940a98f0 2024-03-25T18:19:46.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551324762 0 0) 0x7f1c940b2a30 con 0x7f1c9413f710 2024-03-25T18:19:46.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.438 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:19:46.439 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.436+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94142520 con 0x7f1c9413f710 2024-03-25T18:19:46.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038226312 0 0) 0x7f1c94142520 con 0x7f1c9413f710 2024-03-25T18:19:46.450 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 >> v1:172.21.15.164:6789/0 conn(0x7f1c941430c0 legacy=0x7f1c9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 >> v1:172.21.15.46:6789/0 conn(0x7f1c940a98f0 legacy=0x7f1c940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 monclient: found mon.noname-b 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.46:6790/0 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c940b44c0 con 0x7f1c9413f710 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c84001670 con 0x7f1c9413f710 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c8bfff640 10 monclient: dump: 2024-03-25T18:19:46.451 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c9ba31640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:19:46.452 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.448+0000 7f1c9ba31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:46.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.2 v1:172.21.15.46:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c840017d0 con 0x7f1c9413f710 2024-03-25T18:19:46.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 1 -- 172.21.15.46:0/517687136 <== mon.2 v1:172.21.15.46:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c84001ba0 con 0x7f1c9413f710 2024-03-25T18:19:46.455 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c9ba31640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c9ba31640 4 set_mon_vals no callback set 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.453+0000 7f1c8bfff640 10 monclient: dump: 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:46.456 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 10 monclient: shutdown 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 1 -- 172.21.15.46:0/517687136 >> v1:172.21.15.46:6790/0 conn(0x7f1c9413f710 legacy=0x7f1c941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 1 -- 172.21.15.46:0/517687136 shutdown_connections 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 1 -- 172.21.15.46:0/517687136 >> 172.21.15.46:0/517687136 conn(0x7f1c940b7ea0 msgr2=0x7f1c940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 1 -- 172.21.15.46:0/517687136 shutdown_connections 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.454+0000 7f1c9ba31640 1 -- 172.21.15.46:0/517687136 wait complete. 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 10 monclient: monmap: 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:46.459 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 librados: starting msgr at 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 librados: starting objecter 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 Processor -- start 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 -- start start 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 librados: setting wanted keys 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 1 librados: calling monclient init 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.455+0000 7f1c9ba31640 10 monclient: init 2024-03-25T18:19:46.460 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-b con 0x7f1c940a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-c con 0x7f1c9413f710 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient(hunting): picked mon.noname-a con 0x7f1c941430c0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940ab9e0 con 0x7f1c941430c0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c941d7830 con 0x7f1c940a98f0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c941d7500 con 0x7f1c9413f710 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c9ba31640 10 monclient(hunting): authenticate will time out at 7895.718835s 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c99fce640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c941430c0 0x7f1c94134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57112/0 (socket says 172.21.15.46:57112) 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.456+0000 7f1c99fce640 1 -- 172.21.15.46:0/3564548099 learned_addr learned my addr 172.21.15.46:0/3564548099 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.457+0000 7f1c89ffb640 10 client.?.objecter ms_handle_connect 0x7f1c941430c0 2024-03-25T18:19:46.461 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.457+0000 7f1c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.468+0000 7f1c89ffb640 10 client.?.objecter ms_handle_connect 0x7f1c9413f710 2024-03-25T18:19:46.470 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.468+0000 7f1c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.473+0000 7f1c89ffb640 10 client.?.objecter ms_handle_connect 0x7f1c940a98f0 2024-03-25T18:19:46.474 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.473+0000 7f1c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.499 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:22.417 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496735884 0 0) 0x7f1c941d7500 con 0x7f1c9413f710 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:19:46.500 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.498+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78004eb0 con 0x7f1c9413f710 2024-03-25T18:19:46.506 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:22.423 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.515 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:22.386 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.518 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:22.435 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.521 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:22.390 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.522 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:22.439 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.528 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:22.445 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.539 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:22.410 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.590 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:22.508 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.656 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188585035 0 0) 0x7f1c941d7830 con 0x7f1c940a98f0 2024-03-25T18:19:46.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:19:46.657 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.655+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c941d7500 con 0x7f1c940a98f0 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851431375 0 0) 0x7f1c78004eb0 con 0x7f1c9413f710 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c941d7830 con 0x7f1c9413f710 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c84001ba0 con 0x7f1c9413f710 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:46.662 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:46.663 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:46.663 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.663 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.660+0000 7f1c89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1137332130 0 0) 0x7f1c941d7500 con 0x7f1c940a98f0 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c78004eb0 con 0x7f1c940a98f0 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c90001950 con 0x7f1c940a98f0 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.662+0000 7f1c89ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:46.664 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:46.665 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:46.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2804308314 0 0) 0x7f1c941d7830 con 0x7f1c9413f710 2024-03-25T18:19:46.679 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 >> v1:172.21.15.164:6789/0 conn(0x7f1c940a98f0 legacy=0x7f1c94137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 >> v1:172.21.15.46:6789/0 conn(0x7f1c941430c0 legacy=0x7f1c94134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 1 monclient: found mon.c 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94134790 con 0x7f1c9413f710 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c9ba31640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c9ba31640 10 monclient: _renew_subs 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c9ba31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.677+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c941349e0 con 0x7f1c9413f710 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.678+0000 7f1c9ba31640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:46.680 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.678+0000 7f1c9ba31640 10 monclient: _renew_subs 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.678+0000 7f1c9ba31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.678+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c94138b10 con 0x7f1c9413f710 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.678+0000 7f1c9ba31640 1 librados: init done 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.679+0000 7f1c9ba31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.679+0000 7f1c9ba31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.679+0000 7f1c9ba31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:19:46.681 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:46.679+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c94138dc0 con 0x7f1c9413f710 2024-03-25T18:19:47.007 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c840038a0 con 0x7f1c9413f710 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f1c84003a10 con 0x7f1c9413f710 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c89ffb640 10 monclient: dump: 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.005+0000 7f1c8b7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.006+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f1c840199e0 con 0x7f1c9413f710 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.007+0000 7f1c997cd640 10 monclient: get_auth_request con 0x7f1c7806df20 auth_method 0 2024-03-25T18:19:47.009 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.007+0000 7f1c89ffb640 10 client.4733.objecter ms_handle_connect 0x7f1c7806df20 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.010+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f1c84053590 con 0x7f1c9413f710 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.010+0000 7f1c89ffb640 10 client.4733.objecter ms_dispatch 0x7f1c94000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.010+0000 7f1c89ffb640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.010+0000 7f1c89ffb640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.010+0000 7f1c89ffb640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T18:19:47.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.015+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c8404f0a0 con 0x7f1c9413f710 2024-03-25T18:19:47.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.016+0000 7f1c89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.017 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.016+0000 7f1c89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:47.273 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.271+0000 7f1c9ba31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.271+0000 7f1c9ba31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.271+0000 7f1c9ba31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.46:6790/0 2024-03-25T18:19:47.274 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.271+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 --> v1:172.21.15.46:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c941d7690 con 0x7f1c9413f710 2024-03-25T18:19:47.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.305+0000 7f1c89ffb640 1 -- 172.21.15.46:0/3564548099 <== mon.2 v1:172.21.15.46:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f1c941d7690 con 0x7f1c9413f710 2024-03-25T18:19:47.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.305+0000 7f1c89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.305+0000 7f1c89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:47.307 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:47.307 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 10 librados: watch_flush enter 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 10 librados: watch_flush exit 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 10 client.4733.objecter successfully canceled tick 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 >> v1:172.21.15.164:6800/1612084612 conn(0x7f1c7806df20 legacy=0x7f1c780703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 10 monclient: shutdown 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 >> v1:172.21.15.46:6790/0 conn(0x7f1c9413f710 legacy=0x7f1c941d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 shutdown_connections 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.307+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 >> 172.21.15.46:0/3564548099 conn(0x7f1c940b7ea0 msgr2=0x7f1c9413e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.310 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.308+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 shutdown_connections 2024-03-25T18:19:47.311 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.308+0000 7f1c9ba31640 1 -- 172.21.15.46:0/3564548099 wait complete. 2024-03-25T18:19:47.311 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:47.308+0000 7f1c9ba31640 1 librados: shutdown 2024-03-25T18:19:47.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:51.110 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:27.028 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.118 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:27.035 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.142 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:27.059 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.172 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:27.089 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.175 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:27.092 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.175 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:27.046 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.179 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:27.050 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.204 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:27.075 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:51.208 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:27.125 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.129 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 10 monclient: monmap: 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:19:54.245905+0000 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:19:54.245905+0000 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:54.246 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:54.247 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:19:54.247 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:19:54.247 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 1 Processor -- start 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 1 -- start start 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.244+0000 7f77689d6640 10 monclient: init 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-a con 0x7f77640a98f0 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-c con 0x7f776413f710 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-b con 0x7f77641430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7764142520 con 0x7f77640a98f0 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77640b28a0 con 0x7f776413f710 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77640b2a30 con 0x7f77641430c0 2024-03-25T18:19:54.248 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.249 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.245+0000 7f77689d6640 10 monclient(hunting): authenticate will time out at 7903.508463s 2024-03-25T18:19:54.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.255+0000 7f7761d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f776413f710 0x7f77641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:44104/0 (socket says 172.21.15.46:44104) 2024-03-25T18:19:54.257 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.255+0000 7f7761d9b640 1 -- 172.21.15.46:0/1526871367 learned_addr learned my addr 172.21.15.46:0/1526871367 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:54.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332316783 0 0) 0x7f77640b28a0 con 0x7f776413f710 2024-03-25T18:19:54.267 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:19:54.268 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.266+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774c003520 con 0x7f776413f710 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.275+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244333307 0 0) 0x7f774c003520 con 0x7f776413f710 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.275+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 >> v1:172.21.15.46:6790/0 conn(0x7f77641430c0 legacy=0x7f776414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.275+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 >> v1:172.21.15.46:6789/0 conn(0x7f77640a98f0 legacy=0x7f77640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 1 monclient: found mon.noname-c 2024-03-25T18:19:54.278 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77640b44c0 con 0x7f776413f710 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7758001690 con 0x7f776413f710 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f77689d6640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f77689d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f776159a640 10 monclient: dump: 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:54.279 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.276+0000 7f77689d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77580017f0 con 0x7f776413f710 2024-03-25T18:19:54.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 1 -- 172.21.15.46:0/1526871367 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7758001c00 con 0x7f776413f710 2024-03-25T18:19:54.290 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f77689d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f77689d6640 4 set_mon_vals no callback set 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.288+0000 7f776159a640 10 monclient: dump: 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:54.291 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 10 monclient: shutdown 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 1 -- 172.21.15.46:0/1526871367 >> v1:172.21.15.164:6789/0 conn(0x7f776413f710 legacy=0x7f77641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 1 -- 172.21.15.46:0/1526871367 shutdown_connections 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 1 -- 172.21.15.46:0/1526871367 >> 172.21.15.46:0/1526871367 conn(0x7f77640b7ea0 msgr2=0x7f77640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.289+0000 7f77689d6640 1 -- 172.21.15.46:0/1526871367 shutdown_connections 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.290+0000 7f77689d6640 1 -- 172.21.15.46:0/1526871367 wait complete. 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.290+0000 7f77689d6640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 10 monclient: monmap: 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:19:54.293 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 librados: starting msgr at 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 librados: starting objecter 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 Processor -- start 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 -- start start 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 librados: setting wanted keys 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 1 librados: calling monclient init 2024-03-25T18:19:54.294 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.291+0000 7f77689d6640 10 monclient: init 2024-03-25T18:19:54.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.295 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-b con 0x7f77640a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-a con 0x7f776413f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient(hunting): picked mon.noname-c con 0x7f77641430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7764137df0 con 0x7f776413f710 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77641337a0 con 0x7f77640a98f0 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7764133930 con 0x7f77641430c0 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.296 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.292+0000 7f77689d6640 10 monclient(hunting): authenticate will time out at 7903.555071s 2024-03-25T18:19:54.297 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.295+0000 7f776259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f77640a98f0 0x7f77641392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:44142/0 (socket says 172.21.15.46:44142) 2024-03-25T18:19:54.297 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.296+0000 7f776259c640 1 -- 172.21.15.46:0/1505805192 learned_addr learned my addr 172.21.15.46:0/1505805192 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:19:54.299 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.298+0000 7f7752ffd640 10 client.?.objecter ms_handle_connect 0x7f77640a98f0 2024-03-25T18:19:54.299 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.298+0000 7f7752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.303+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716458354 0 0) 0x7f77641337a0 con 0x7f77640a98f0 2024-03-25T18:19:54.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.304+0000 7f7752ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.304+0000 7f7752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.304+0000 7f7752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.305 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.304+0000 7f7752ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:19:54.306 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.304+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7744004eb0 con 0x7f77640a98f0 2024-03-25T18:19:54.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.305+0000 7f7752ffd640 10 client.?.objecter ms_handle_connect 0x7f77641430c0 2024-03-25T18:19:54.307 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.305+0000 7f7752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.312+0000 7f7752ffd640 10 client.?.objecter ms_handle_connect 0x7f776413f710 2024-03-25T18:19:54.314 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.312+0000 7f7752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923930546 0 0) 0x7f7764133930 con 0x7f77641430c0 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:19:54.318 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.316+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77641337a0 con 0x7f77641430c0 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262296307 0 0) 0x7f7744004eb0 con 0x7f77640a98f0 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7764133930 con 0x7f77640a98f0 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7754001a30 con 0x7f77640a98f0 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.319 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.317+0000 7f7752ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:54.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359836385 0 0) 0x7f7764133930 con 0x7f77640a98f0 2024-03-25T18:19:54.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 >> v1:172.21.15.46:6790/0 conn(0x7f77641430c0 legacy=0x7f7764133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.325 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 >> v1:172.21.15.46:6789/0 conn(0x7f776413f710 legacy=0x7f77641376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 1 monclient: found mon.b 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77641367d0 con 0x7f77640a98f0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f7752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 10 monclient: _renew_subs 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77641369e0 con 0x7f77640a98f0 2024-03-25T18:19:54.326 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 10 monclient: _renew_subs 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776413b640 con 0x7f77640a98f0 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.323+0000 7f77689d6640 1 librados: init done 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.324+0000 7f77689d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.324+0000 7f77689d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.327 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.324+0000 7f77689d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.328 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.324+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77640627b0 con 0x7f77640a98f0 2024-03-25T18:19:54.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.332+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7754001c70 con 0x7f77640a98f0 2024-03-25T18:19:54.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.332+0000 7f7752ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.334 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.332+0000 7f7760d99640 4 set_mon_vals no callback set 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.599+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f7754003a60 con 0x7f77640a98f0 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.599+0000 7f7752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.599+0000 7f7752ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.599+0000 7f7752ffd640 10 monclient: dump: 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:19:54.601 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:19:54.602 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:19:54.602 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:19:54.602 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.599+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f7754003c40 con 0x7f77640a98f0 2024-03-25T18:19:54.602 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.600+0000 7f7761d9b640 10 monclient: get_auth_request con 0x7f774406dcc0 auth_method 0 2024-03-25T18:19:54.602 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.600+0000 7f7752ffd640 10 client.4723.objecter ms_handle_connect 0x7f774406dcc0 2024-03-25T18:19:54.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.602+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f7754054900 con 0x7f77640a98f0 2024-03-25T18:19:54.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.602+0000 7f7752ffd640 10 client.4723.objecter ms_dispatch 0x7f7764000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:54.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.602+0000 7f7752ffd640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:54.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.602+0000 7f7752ffd640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:54.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.602+0000 7f7752ffd640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T18:19:54.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.605+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7754050410 con 0x7f77640a98f0 2024-03-25T18:19:54.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.605+0000 7f7752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.605+0000 7f7752ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:54.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.831+0000 7f77689d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.831+0000 7f77689d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.831+0000 7f77689d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:19:54.833 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.831+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7764137300 con 0x7f77640a98f0 2024-03-25T18:19:54.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.861+0000 7f7752ffd640 1 -- 172.21.15.46:0/1505805192 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f7754052340 con 0x7f77640a98f0 2024-03-25T18:19:54.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.861+0000 7f7752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.863 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.861+0000 7f7752ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:54.863 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:19:54.864 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:54.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 10 librados: watch_flush enter 2024-03-25T18:19:54.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 10 librados: watch_flush exit 2024-03-25T18:19:54.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T18:19:54.866 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 10 client.4723.objecter successfully canceled tick 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 >> v1:172.21.15.164:6800/1612084612 conn(0x7f774406dcc0 legacy=0x7f7744070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 10 monclient: shutdown 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 >> v1:172.21.15.164:6789/0 conn(0x7f77640a98f0 legacy=0x7f77641392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 shutdown_connections 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 >> 172.21.15.46:0/1505805192 conn(0x7f77640b7ea0 msgr2=0x7f7764074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 shutdown_connections 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 -- 172.21.15.46:0/1505805192 wait complete. 2024-03-25T18:19:54.867 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:19:54.864+0000 7f77689d6640 1 librados: shutdown 2024-03-25T18:19:54.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:01.683 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.798+0000 7fe97ccaf640 10 monclient: get_monmap_and_config 2024-03-25T18:20:01.800 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.798+0000 7fe97ccaf640 10 monclient: build_initial_monmap 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.798+0000 7fe97ccaf640 10 monclient: monmap: 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:20:01.800391+0000 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:20:01.800391+0000 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:20:01.801 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.799+0000 7fe97ccaf640 1 Processor -- start 2024-03-25T18:20:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.799+0000 7fe97ccaf640 1 -- start start 2024-03-25T18:20:01.802 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.799+0000 7fe97ccaf640 10 monclient: init 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe9780a98f0 addr v1:172.21.15.46:6790/0 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe97813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe9781430c0 addr v1:172.21.15.164:6789/0 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978142520 con 0x7fe97813f710 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780b28a0 con 0x7fe9781430c0 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780b2a30 con 0x7fe9780a98f0 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:01.803 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.800+0000 7fe97ccaf640 10 monclient(hunting): authenticate will time out at 7911.063471s 2024-03-25T18:20:01.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.811+0000 7fe976d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe9781430c0 0x7fe97814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:48980/0 (socket says 172.21.15.46:48980) 2024-03-25T18:20:01.812 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.811+0000 7fe976d9d640 1 -- 172.21.15.46:0/2470694916 learned_addr learned my addr 172.21.15.46:0/2470694916 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193164823 0 0) 0x7fe9780b28a0 con 0x7fe9781430c0 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:20:01.822 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.820+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe960003520 con 0x7fe9781430c0 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122207916 0 0) 0x7fe9780b2a30 con 0x7fe9780a98f0 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:20:01.835 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.833+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9780b28a0 con 0x7fe9780a98f0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809926843 0 0) 0x7fe960003520 con 0x7fe9781430c0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 >> v1:172.21.15.46:6790/0 conn(0x7fe9780a98f0 legacy=0x7fe9780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 >> v1:172.21.15.46:6789/0 conn(0x7fe97813f710 legacy=0x7fe9781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 monclient: found mon.noname-c 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780b44c0 con 0x7fe9781430c0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 10 monclient: _finish_auth 0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe970001a10 con 0x7fe9781430c0 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97ccaf640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:20:01.840 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97ccaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.837+0000 7fe97559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.838+0000 7fe97559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.838+0000 7fe97559a640 10 monclient: dump: 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:01.841 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.838+0000 7fe97ccaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe970001b70 con 0x7fe9781430c0 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 1 -- 172.21.15.46:0/2470694916 <== mon.1 v1:172.21.15.164:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe970001f80 con 0x7fe9781430c0 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97559a640 10 monclient: dump: 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:01.856 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97ccaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:01.857 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.854+0000 7fe97ccaf640 4 set_mon_vals no callback set 2024-03-25T18:20:01.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 10 monclient: shutdown 2024-03-25T18:20:01.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:01.858 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2470694916 >> v1:172.21.15.164:6789/0 conn(0x7fe9781430c0 legacy=0x7fe97814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2470694916 shutdown_connections 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2470694916 >> 172.21.15.46:0/2470694916 conn(0x7fe9780b7ea0 msgr2=0x7fe9780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2470694916 shutdown_connections 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.855+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2470694916 wait complete. 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.856+0000 7fe97ccaf640 10 monclient: build_initial_monmap 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.856+0000 7fe97ccaf640 10 monclient: monmap: 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:20:01.859 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:20:01.860 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:01.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 librados: starting msgr at 2024-03-25T18:20:01.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 librados: starting objecter 2024-03-25T18:20:01.860 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 Processor -- start 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 -- start start 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 librados: setting wanted keys 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 1 librados: calling monclient init 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.857+0000 7fe97ccaf640 10 monclient: init 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe9780a98f0 addr v1:172.21.15.164:6789/0 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe97813f710 addr v1:172.21.15.46:6789/0 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe9781430c0 addr v1:172.21.15.46:6790/0 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978136ec0 con 0x7fe97813f710 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978136c20 con 0x7fe9780a98f0 2024-03-25T18:20:01.861 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978137db0 con 0x7fe9781430c0 2024-03-25T18:20:01.862 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:01.862 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.858+0000 7fe97ccaf640 10 monclient(hunting): authenticate will time out at 7911.121170s 2024-03-25T18:20:01.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.872+0000 7fe97659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe9780a98f0 0x7fe9781376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:49002/0 (socket says 172.21.15.46:49002) 2024-03-25T18:20:01.874 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.872+0000 7fe97659c640 1 -- 172.21.15.46:0/2201539846 learned_addr learned my addr 172.21.15.46:0/2201539846 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:20:01.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.874+0000 7fe966ffd640 10 client.?.objecter ms_handle_connect 0x7fe9780a98f0 2024-03-25T18:20:01.876 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.874+0000 7fe966ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.879+0000 7fe966ffd640 10 client.?.objecter ms_handle_connect 0x7fe97813f710 2024-03-25T18:20:01.881 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.879+0000 7fe966ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.880+0000 7fe966ffd640 10 client.?.objecter ms_handle_connect 0x7fe9781430c0 2024-03-25T18:20:01.882 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.880+0000 7fe966ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167431817 0 0) 0x7fe978136ec0 con 0x7fe97813f710 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:20:01.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.889+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe958004eb0 con 0x7fe97813f710 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111453316 0 0) 0x7fe978136c20 con 0x7fe9780a98f0 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:20:01.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.893+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe978136ec0 con 0x7fe9780a98f0 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179341940 0 0) 0x7fe978137db0 con 0x7fe9781430c0 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:20:01.904 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.902+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe978136c20 con 0x7fe9781430c0 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 709958713 0 0) 0x7fe958004eb0 con 0x7fe97813f710 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe978137db0 con 0x7fe97813f710 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe96c001930 con 0x7fe97813f710 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:01.914 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:01.915 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.912+0000 7fe966ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:01.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267881583 0 0) 0x7fe978137db0 con 0x7fe97813f710 2024-03-25T18:20:01.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 >> v1:172.21.15.46:6790/0 conn(0x7fe9781430c0 legacy=0x7fe9781e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 >> v1:172.21.15.164:6789/0 conn(0x7fe9780a98f0 legacy=0x7fe9781376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.927 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 1 monclient: found mon.a 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978138b50 con 0x7fe97813f710 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 10 monclient: _renew_subs 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe978139830 con 0x7fe97813f710 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 10 monclient: _renew_subs 2024-03-25T18:20:01.928 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9781369b0 con 0x7fe97813f710 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.925+0000 7fe97ccaf640 1 librados: init done 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.926+0000 7fe97ccaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.926+0000 7fe97ccaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.926+0000 7fe97ccaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:20:01.929 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:01.926+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe978139a20 con 0x7fe97813f710 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe96c003610 con 0x7fe97813f710 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7fe96c003800 con 0x7fe97813f710 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 10 monclient: dump: 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe974d99640 4 set_mon_vals no callback set 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.248+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7fe96c003a20 con 0x7fe97813f710 2024-03-25T18:20:02.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.249+0000 7fe97659c640 10 monclient: get_auth_request con 0x7fe95806dc00 auth_method 0 2024-03-25T18:20:02.252 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.249+0000 7fe966ffd640 10 client.4728.objecter ms_handle_connect 0x7fe95806dc00 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7fe96c0534c0 con 0x7fe97813f710 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 10 client.4728.objecter ms_dispatch 0x7fe978000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe96c04efd0 con 0x7fe97813f710 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.258 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.256+0000 7fe966ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:02.493 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.491+0000 7fe97ccaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.491+0000 7fe97ccaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.491+0000 7fe97ccaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.46:6789/0 2024-03-25T18:20:02.494 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.491+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 --> v1:172.21.15.46:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9780627b0 con 0x7fe97813f710 2024-03-25T18:20:02.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.574+0000 7fe966ffd640 1 -- 172.21.15.46:0/2201539846 <== mon.0 v1:172.21.15.46:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7fe96c050f00 con 0x7fe97813f710 2024-03-25T18:20:02.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.574+0000 7fe966ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.576 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.574+0000 7fe966ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:02.576 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:20:02.577 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 10 librados: watch_flush enter 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 10 librados: watch_flush exit 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 10 client.4728.objecter successfully canceled tick 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 >> v1:172.21.15.164:6800/1612084612 conn(0x7fe95806dc00 legacy=0x7fe9580700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 10 monclient: shutdown 2024-03-25T18:20:02.579 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 >> v1:172.21.15.46:6789/0 conn(0x7fe97813f710 legacy=0x7fe9780b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 shutdown_connections 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 >> 172.21.15.46:0/2201539846 conn(0x7fe9780b7ea0 msgr2=0x7fe97813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 shutdown_connections 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 -- 172.21.15.46:0/2201539846 wait complete. 2024-03-25T18:20:02.580 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:02.577+0000 7fe97ccaf640 1 librados: shutdown 2024-03-25T18:20:02.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:09.398 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:09.515 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.514+0000 7f2d2a8d4640 10 monclient: get_monmap_and_config 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.514+0000 7f2d2a8d4640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.514+0000 7f2d2a8d4640 10 monclient: monmap: 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:20:09.515860+0000 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:20:09.515860+0000 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-c 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-b 2024-03-25T18:20:09.516 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:09.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.515+0000 7f2d2a8d4640 1 Processor -- start 2024-03-25T18:20:09.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.515+0000 7f2d2a8d4640 1 -- start start 2024-03-25T18:20:09.517 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.515+0000 7f2d2a8d4640 10 monclient: init 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-c con 0x7f2d240b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2d240b1730 addr v1:172.21.15.46:6790/0 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2414be90 addr v1:172.21.15.46:6789/0 2024-03-25T18:20:09.518 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240af940 con 0x7f2d2414be90 2024-03-25T18:20:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240b0a70 con 0x7f2d240b4d60 2024-03-25T18:20:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240b0d50 con 0x7f2d240b1730 2024-03-25T18:20:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.519 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.516+0000 7f2d2a8d4640 10 monclient(hunting): authenticate will time out at 7918.779259s 2024-03-25T18:20:09.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.530+0000 7f2d23fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2d240b4d60 0x7f2d2413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:55240/0 (socket says 172.21.15.46:55240) 2024-03-25T18:20:09.532 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.530+0000 7f2d23fff640 1 -- 172.21.15.46:0/3134257826 learned_addr learned my addr 172.21.15.46:0/3134257826 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:20:09.535 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781490130 0 0) 0x7f2d240b0a70 con 0x7f2d240b4d60 2024-03-25T18:20:09.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:20:09.536 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.534+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c003520 con 0x7f2d240b4d60 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505675020 0 0) 0x7f2d0c003520 con 0x7f2d240b4d60 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 >> v1:172.21.15.46:6790/0 conn(0x7f2d240b1730 legacy=0x7f2d240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 >> v1:172.21.15.46:6789/0 conn(0x7f2d2414be90 legacy=0x7f2d2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 monclient: found mon.noname-c 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.164:6789/0 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240b1030 con 0x7f2d240b4d60 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.545 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2d14001690 con 0x7f2d240b4d60 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d22ffd640 10 monclient: dump: 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:09.546 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d2a8d4640 5 monclient: authenticate success, global_id 4732 2024-03-25T18:20:09.547 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.543+0000 7f2d2a8d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:09.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.550+0000 7f2d22ffd640 1 -- 172.21.15.46:0/3134257826 <== mon.1 v1:172.21.15.164:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d140017f0 con 0x7f2d240b4d60 2024-03-25T18:20:09.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.550+0000 7f2d22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.550+0000 7f2d2a8d4640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:09.552 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.550+0000 7f2d2a8d4640 4 set_mon_vals no callback set 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 10 monclient: shutdown 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3134257826 >> v1:172.21.15.164:6789/0 conn(0x7f2d240b4d60 legacy=0x7f2d2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3134257826 shutdown_connections 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3134257826 >> 172.21.15.46:0/3134257826 conn(0x7f2d240617a0 msgr2=0x7f2d24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3134257826 shutdown_connections 2024-03-25T18:20:09.553 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.551+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3134257826 wait complete. 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 10 monclient: monmap: 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:epoch 0 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:last_changed 0.000000 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:created 0.000000 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 1 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.noname-a 2024-03-25T18:20:09.554 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.noname-b 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.noname-c 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 librados: starting msgr at 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 librados: starting objecter 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 Processor -- start 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 -- start start 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 librados: setting wanted keys 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 1 librados: calling monclient init 2024-03-25T18:20:09.555 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.552+0000 7f2d2a8d4640 10 monclient: init 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2d240b1730 addr v1:172.21.15.46:6789/0 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2d240b4d60 addr v1:172.21.15.164:6789/0 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2414be90 addr v1:172.21.15.46:6790/0 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d241d8620 con 0x7f2d240b1730 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d241d79f0 con 0x7f2d240b4d60 2024-03-25T18:20:09.556 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 1 -- --> v1:172.21.15.46:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2413b080 con 0x7f2d2414be90 2024-03-25T18:20:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.557 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.553+0000 7f2d2a8d4640 10 monclient(hunting): authenticate will time out at 7918.816139s 2024-03-25T18:20:09.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.558+0000 7f2d237fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2d240b4d60 0x7f2d24142900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.46:55268/0 (socket says 172.21.15.46:55268) 2024-03-25T18:20:09.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.558+0000 7f2d237fe640 1 -- 172.21.15.46:0/3032658832 learned_addr learned my addr 172.21.15.46:0/3032658832 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-03-25T18:20:09.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.559+0000 7f2d20ff9640 10 client.?.objecter ms_handle_connect 0x7f2d240b4d60 2024-03-25T18:20:09.560 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.559+0000 7f2d20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702446965 0 0) 0x7f2d241d79f0 con 0x7f2d240b4d60 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:20:09.575 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.573+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10004eb0 con 0x7f2d240b4d60 2024-03-25T18:20:09.581 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.580+0000 7f2d20ff9640 10 client.?.objecter ms_handle_connect 0x7f2d240b1730 2024-03-25T18:20:09.582 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.580+0000 7f2d20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.582+0000 7f2d20ff9640 10 client.?.objecter ms_handle_connect 0x7f2d2414be90 2024-03-25T18:20:09.583 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.582+0000 7f2d20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800283256 0 0) 0x7f2d241d8620 con 0x7f2d240b1730 2024-03-25T18:20:09.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.593 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:20:09.594 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.591+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d241d79f0 con 0x7f2d240b1730 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228943765 0 0) 0x7f2d10004eb0 con 0x7f2d240b4d60 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d241d8620 con 0x7f2d240b4d60 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2d18001a30 con 0x7f2d240b4d60 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.595 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:09.596 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:09.596 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:09.596 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:09.596 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:09.596 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.594+0000 7f2d20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.2 v1:172.21.15.46:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184831226 0 0) 0x7f2d2413b080 con 0x7f2d2414be90 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 10 monclient(hunting): my global_id is 4751 2024-03-25T18:20:09.597 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.595+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.46:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10004eb0 con 0x7f2d2414be90 2024-03-25T18:20:09.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256892766 0 0) 0x7f2d241d8620 con 0x7f2d240b4d60 2024-03-25T18:20:09.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 >> v1:172.21.15.46:6790/0 conn(0x7f2d2414be90 legacy=0x7f2d241e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.604 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 >> v1:172.21.15.46:6789/0 conn(0x7f2d240b1730 legacy=0x7f2d241d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 1 monclient: found mon.b 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2413b210 con 0x7f2d240b4d60 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 10 monclient: _renew_subs 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d241405f0 con 0x7f2d240b4d60 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 10 monclient: _renew_subs 2024-03-25T18:20:09.605 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:20:09.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2413ba20 con 0x7f2d240b4d60 2024-03-25T18:20:09.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.602+0000 7f2d2a8d4640 1 librados: init done 2024-03-25T18:20:09.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.603+0000 7f2d2a8d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.603+0000 7f2d2a8d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.606 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.603+0000 7f2d2a8d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:20:09.607 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.603+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d241d7b80 con 0x7f2d240b4d60 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d18001c70 con 0x7f2d240b4d60 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2154860171 0 0) 0x7f2d18003a60 con 0x7f2d240b4d60 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:09.890 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d20ff9640 10 monclient: dump: 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:epoch 1 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:fsid a6bb6b92-3bc5-4b76-b609-dd01555b77cc 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:last_changed 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:created 2024-03-25T18:07:53.128179+0000 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:election_strategy: 3 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:0: v1:172.21.15.46:6789/0 mon.a 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:1: v1:172.21.15.164:6789/0 mon.b 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:2: v1:172.21.15.46:6790/0 mon.c 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr: 2024-03-25T18:20:09.891 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.888+0000 7f2d227fc640 4 set_mon_vals no callback set 2024-03-25T18:20:09.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.889+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1134679645 0 0) 0x7f2d18003c40 con 0x7f2d240b4d60 2024-03-25T18:20:09.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.890+0000 7f2d23fff640 10 monclient: get_auth_request con 0x7f2d1006de00 auth_method 0 2024-03-25T18:20:09.892 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.890+0000 7f2d20ff9640 10 client.4735.objecter ms_handle_connect 0x7f2d1006de00 2024-03-25T18:20:09.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.891+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1627265648 0 0) 0x7f2d18054850 con 0x7f2d240b4d60 2024-03-25T18:20:09.893 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.891+0000 7f2d20ff9640 10 client.4735.objecter ms_dispatch 0x7f2d24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:09.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.891+0000 7f2d20ff9640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:09.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.891+0000 7f2d20ff9640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:09.894 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.891+0000 7f2d20ff9640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T18:20:09.895 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.894+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d18050360 con 0x7f2d240b4d60 2024-03-25T18:20:09.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.894+0000 7f2d20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.896 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:09.894+0000 7f2d20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.125+0000 7f2d2a8d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.125+0000 7f2d2a8d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.125+0000 7f2d2a8d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.164:6789/0 2024-03-25T18:20:10.127 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.125+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 --> v1:172.21.15.164:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d240627b0 con 0x7f2d240b4d60 2024-03-25T18:20:10.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.315+0000 7f2d20ff9640 1 -- 172.21.15.46:0/3032658832 <== mon.1 v1:172.21.15.164:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4162761605) 0x7f2d18052290 con 0x7f2d240b4d60 2024-03-25T18:20:10.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.315+0000 7f2d20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.317 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.315+0000 7f2d20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:10.317 INFO:teuthology.misc.health.smithi046.stdout: 2024-03-25T18:20:10.317 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":11,"fsid":"a6bb6b92-3bc5-4b76-b609-dd01555b77cc","created":"2024-03-25T18:08:40.636652+0000","modified":"2024-03-25T18:18:45.686854+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:18:45.686854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5134f463-b5c2-47f0-9dff-01724449ce31","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1a2f7f36-cac7-4c85-a49c-9d6ba1e31c59","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e2dc8ca4-71c2-4924-b4ba-fa7cbeeb2406","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4f499a38-37ef-4a56-8e81-2f8b72374926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0b1ce7be-c1d6-4261-8986-44f609df1cbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f362b74d-5f3b-4df8-bbe5-687f796063c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e8470de7-cbc1-44c0-8b6d-bb7d71044b72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e449439f-9f64-42ae-aecc-87980c327dd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:10.319 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 10 librados: watch_flush enter 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 10 librados: watch_flush exit 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 10 client.4735.objecter successfully canceled tick 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 >> v1:172.21.15.164:6800/1612084612 conn(0x7f2d1006de00 legacy=0x7f2d100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 10 monclient: shutdown 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 >> v1:172.21.15.164:6789/0 conn(0x7f2d240b4d60 legacy=0x7f2d24142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 shutdown_connections 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 >> 172.21.15.46:0/3032658832 conn(0x7f2d240617a0 msgr2=0x7f2d2413c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 shutdown_connections 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 -- 172.21.15.46:0/3032658832 wait complete. 2024-03-25T18:20:10.320 INFO:teuthology.misc.health.smithi046.stderr:2024-03-25T18:20:10.317+0000 7f2d2a8d4640 1 librados: shutdown 2024-03-25T18:20:10.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:10.331 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:20:10.333 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:20:10.333 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:20:10.333 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:11.168 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:47.086 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.174 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:47.091 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.229 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:47.146 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.229 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:11:47.147 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.241 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:47.159 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.251 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:47.122 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.257 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:47.128 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.270 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:11:47.188 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:11.284 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:11:47.155 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.370 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:11:25.847 35792== 2024-03-25T18:20:12.370 INFO:tasks.ceph.osd.0.smithi046.stderr:==00:00:11:25.847 35792== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:12.402 INFO:tasks.ceph.osd.0.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:16.435 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:20:16.436 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:20:16.436 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:20:16.436 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:18.497 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:11:31.974 35794== 2024-03-25T18:20:18.497 INFO:tasks.ceph.osd.1.smithi046.stderr:==00:00:11:31.974 35794== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:18.532 INFO:tasks.ceph.osd.1.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:22.538 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:20:22.539 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:20:22.539 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:20:22.539 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:24.509 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:11:37.970 35796== 2024-03-25T18:20:24.509 INFO:tasks.ceph.osd.2.smithi046.stderr:==00:00:11:37.970 35796== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:24.546 INFO:tasks.ceph.osd.2.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:28.641 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:20:28.642 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:20:28.642 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:20:28.642 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:30.551 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:11:44.010 35797== 2024-03-25T18:20:30.551 INFO:tasks.ceph.osd.3.smithi046.stderr:==00:00:11:44.010 35797== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:30.589 INFO:tasks.ceph.osd.3.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:31.149 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:07.067 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.202 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:07.120 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.239 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:07.156 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.249 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:07.120 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.259 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:07.130 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.272 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:07.189 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.276 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:07.194 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.282 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:07.153 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:31.295 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:07.212 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:34.744 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:20:34.745 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:20:34.745 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:20:34.745 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:36.817 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:11:50.265 35502== 2024-03-25T18:20:36.817 INFO:tasks.ceph.osd.4.smithi164.stderr:==00:00:11:50.265 35502== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:36.858 INFO:tasks.ceph.osd.4.smithi164.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:40.847 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:20:40.848 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:20:40.848 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:20:40.848 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:42.822 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:11:56.286 35500== 2024-03-25T18:20:42.822 INFO:tasks.ceph.osd.5.smithi164.stderr:==00:00:11:56.286 35500== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:42.858 INFO:tasks.ceph.osd.5.smithi164.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:46.950 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:20:46.951 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:20:46.951 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:20:46.951 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:48.636 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:12:02.100 35501== 2024-03-25T18:20:48.637 INFO:tasks.ceph.osd.6.smithi164.stderr:==00:00:12:02.100 35501== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:48.682 INFO:tasks.ceph.osd.6.smithi164.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:51.238 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:27.155 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.240 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:27.158 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.256 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:27.174 35485== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.286 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:27.157 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.287 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:27.158 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.296 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:27.214 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.298 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:27.216 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.300 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:27.171 34957== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.313 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:27.230 35482== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:53.053 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:20:53.053 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:20:53.053 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:20:53.054 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:54.633 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:12:08.074 35503== 2024-03-25T18:20:54.633 INFO:tasks.ceph.osd.7.smithi164.stderr:==00:00:12:08.074 35503== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:54.664 INFO:tasks.ceph.osd.7.smithi164.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:59.155 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:20:59.156 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:20:59.156 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:20:59.156 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:20:59.156 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:59.183 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:20:59.183 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:20:59.184 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:59.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:20:59.184+0000 7f9d60800640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34955) UID: 0 2024-03-25T18:20:59.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-25T18:20:59.184+0000 7f9d60800640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:20:59.213 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:20:59.213 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:20:59.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:20:59.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:59.239 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-03-25T18:20:59.227+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35448) UID: 0 2024-03-25T18:20:59.240 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-03-25T18:20:59.229+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:20:59.662 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:35.580 35482== 2024-03-25T18:20:59.663 INFO:tasks.ceph.mon.a.smithi046.stderr:==00:00:12:35.580 35482== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:59.672 INFO:tasks.ceph.mon.a.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:05.315 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:21:05.315 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:21:05.316 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:21:05.316 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:05.334 INFO:tasks.ceph.mon.c.smithi046.stderr:2024-03-25T18:21:05.319+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35450) UID: 0 2024-03-25T18:21:05.334 INFO:tasks.ceph.mon.c.smithi046.stderr:2024-03-25T18:21:05.321+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:21:05.838 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:41.755 35485== 2024-03-25T18:21:05.838 INFO:tasks.ceph.mon.c.smithi046.stderr:==00:00:12:41.755 35485== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:05.848 INFO:tasks.ceph.mon.c.smithi046.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:11.417 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:21:11.418 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:21:11.418 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:21:11.418 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:11.434 INFO:tasks.ceph.mon.b.smithi164.stderr:2024-03-25T18:21:11.421+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34953) UID: 0 2024-03-25T18:21:11.434 INFO:tasks.ceph.mon.b.smithi164.stderr:2024-03-25T18:21:11.422+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:21:11.968 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:47.838 34957== 2024-03-25T18:21:11.968 INFO:tasks.ceph.mon.b.smithi164.stderr:==00:00:12:47.838 34957== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:11.977 INFO:tasks.ceph.mon.b.smithi164.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:17.519 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:21:17.520 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:21:17.520 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:21:17.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:21:17.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:21:17.571 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:21:17.691 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:21:17.691 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:21:17.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:21:17.757 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:21:17.815 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:21:17.815 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:21:17.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:21:17.890 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:21:18.061 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:21:18.061 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:21:18.133 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:21:18.133 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:21:18.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:21:18.207 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:21:18.266 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:21:18.269 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/data/mon.a.tgz 2024-03-25T18:21:18.269 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-03-25T18:21:18.285 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.Lv9wQJlQ6t 2024-03-25T18:21:18.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Lv9wQJlQ6t 2024-03-25T18:21:18.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.Lv9wQJlQ6t 2024-03-25T18:21:18.476 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.Lv9wQJlQ6t is 133KB 2024-03-25T18:21:18.528 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.Lv9wQJlQ6t 2024-03-25T18:21:18.542 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/data/mon.c.tgz 2024-03-25T18:21:18.543 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-03-25T18:21:18.598 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.dOEwdU6Trh 2024-03-25T18:21:18.598 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dOEwdU6Trh 2024-03-25T18:21:18.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.dOEwdU6Trh 2024-03-25T18:21:18.771 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.dOEwdU6Trh is 133KB 2024-03-25T18:21:18.821 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.dOEwdU6Trh 2024-03-25T18:21:18.835 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/data/mon.b.tgz 2024-03-25T18:21:18.835 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-25T18:21:18.853 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.NRC5T6hsqI 2024-03-25T18:21:18.853 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NRC5T6hsqI 2024-03-25T18:21:18.957 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.NRC5T6hsqI 2024-03-25T18:21:19.043 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.NRC5T6hsqI is 133KB 2024-03-25T18:21:19.094 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.NRC5T6hsqI 2024-03-25T18:21:19.109 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:21:19.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:21:19.111 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:21:19.467 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:19.708 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:21:19.710 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi046/crash 2024-03-25T18:21:19.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:21:19.738 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi164/crash 2024-03-25T18:21:19.739 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:21:19.765 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:21:19.765 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:21:19.783 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42390.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42039.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42390.log: /var/log/ceph/ceph-client.admin.42039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41376.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42390.log.gz 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42039.log.gz 2024-03-25T18:21:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41376.log: gzip -5 --verbose -- 86.4% /var/log/ceph/ceph-client.admin.39815.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41376.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39893.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41805.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39815.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42000.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40751.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41805.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr: 86.1%/var/log/ceph/ceph-client.admin.42000.log: -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-03-25T18:21:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42000.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35820.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35490.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-03-25T18:21:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35490.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35820.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39502.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40400.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35490.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41649.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39502.log: /var/log/ceph/ceph-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39502.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39659.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41649.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41298.log 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-25T18:21:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36862.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41688.log: /var/log/ceph/ceph-client.admin.41298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.36862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41298.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36862.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-03-25T18:21:19.859 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42546.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42546.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41454.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 86.1%/var/log/ceph/ceph-client.admin.40829.log: -- replaced with /var/log/ceph/ceph-client.admin.42546.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41337.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41454.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37907.log 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-25T18:21:19.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41337.log: gzip 86.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph.tmp-client.admin.29624.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42078.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41337.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.29624.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.37907.log.gz --verbose 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.40439.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29624.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42078.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39854.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35795.log 2024-03-25T18:21:19.861 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41922.log: gzip -5 --verbose 87.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz /var/log/ceph/ceph-osd.1.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35795.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41415.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42234.log: gzip -5 --verbose -- 86.3% /var/log/ceph/ceph-client.admin.40985.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41415.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41415.log.gz 2024-03-25T18:21:19.862 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.40361.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40985.log: /var/log/ceph/ceph-client.admin.39698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40361.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41532.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41532.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41181.log 2024-03-25T18:21:19.863 INFO:teuthology.orchestra.run.smithi046.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41532.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42429.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42273.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42429.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41181.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42273.log: /var/log/ceph/ceph-client.admin.42195.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42429.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42195.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42273.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41493.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41844.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42195.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2024-03-25T18:21:19.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39344.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41493.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40322.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40088.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41063.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr: 86.1%/var/log/ceph/ceph-client.admin.41063.log: -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41063.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40673.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40556.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-25T18:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42468.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39383.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41102.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42468.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.42117.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39383.log: gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39383.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42117.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42117.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41727.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: gzip 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-03-25T18:21:19.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42585.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40244.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41727.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 87.4% -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42585.log: /var/log/ceph/ceph-client.admin.40166.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42585.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41610.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42351.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41610.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41610.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41220.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42351.log: /var/log/ceph/ceph-client.admin.41220.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42351.log.gz 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-25T18:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41220.log.gz 2024-03-25T18:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42312.log 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40595.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42156.log 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41571.log 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42312.log: /var/log/ceph/ceph-client.admin.42156.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42312.log.gz 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-03-25T18:21:19.869 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42156.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41571.log: /var/log/ceph/ceph-client.admin.41024.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35981.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39620.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.39971.log: 86.2% 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39541.log 2024-03-25T18:21:19.870 INFO:teuthology.orchestra.run.smithi046.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-25T18:21:19.871 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.39541.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39541.log.gz 2024-03-25T18:21:19.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35401.log 2024-03-25T18:21:19.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35270.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35401.log: /var/log/ceph/ceph-client.admin.35270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35401.log.gz 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35270.log.gz 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:21:19.872 INFO:teuthology.orchestra.run.smithi164.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35230.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35230.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35310.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35230.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:21:19.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35352.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35110.log 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35110.log: /var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:21:19.874 INFO:teuthology.orchestra.run.smithi164.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-03-25T18:21:19.922 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:21:20.640 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:21:20.642 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:21:20.676 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:21:20.724 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:21:20.758 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:21:20.946 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:21:20.959 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:21:21.010 INFO:teuthology.orchestra.run.smithi164.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:21:25.008 INFO:teuthology.orchestra.run.smithi164.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:21:25.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-25T18:21:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m5.183s 2024-03-25T18:21:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m8.732s 2024-03-25T18:21:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.336s 2024-03-25T18:21:25.123 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:21:25.344 INFO:teuthology.orchestra.run.smithi046.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:21:25.346 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-25T18:21:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m5.551s 2024-03-25T18:21:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m14.277s 2024-03-25T18:21:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.528s 2024-03-25T18:21:25.346 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:21:25.346 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi046/log 2024-03-25T18:21:25.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:21:27.740 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi164/log 2024-03-25T18:21:27.741 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:21:29.175 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:29.381 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ba13d80b99034eb7b47f79a369718768 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:21:29.385 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:21:29.394 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:21:29.428 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:21:29.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:21:29.443 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:21:29.525 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:21:29.537 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:21:29.538 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:21:29.540 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:21:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:21:29.559 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:21:29.559 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-25T18:21:29.559 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +592us[ +592us] +/- 96ms 2024-03-25T18:21:29.560 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1286us[-1279us] +/- 77ms 2024-03-25T18:21:29.560 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +752us[ +752us] +/- 66ms 2024-03-25T18:21:29.560 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:21:29.561 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:21:29.562 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-25T18:21:29.562 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +570us[ +559us] +/- 96ms 2024-03-25T18:21:29.562 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1317us[-1329us] +/- 77ms 2024-03-25T18:21:29.562 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +729us[ +729us] +/- 66ms 2024-03-25T18:21:29.562 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:21:29.562 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:21:29.575 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:21:29.575 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:21:29.591 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:21:29.674 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:21:29.747 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:21:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:29.779 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:21:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:29.812 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:21:29.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:21:29.837 INFO:teuthology.task.internal:Duration was 7675.992205 seconds 2024-03-25T18:21:29.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:21:29.852 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:21:29.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:21:29.854 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:21:29.895 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:21:29.899 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:21:30.088 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:21:30.088 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-03-25T18:21:30.089 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:21:30.114 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-03-25T18:21:30.114 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:21:30.136 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:21:30.136 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:21:30.156 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:21:30.205 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:21:30.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:21:30.226 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:21:30.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:21:30.360 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:21:30.361 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:21:30.387 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:21:30.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:21:30.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:30.431 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:30.453 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-03-25T18:21:30.480 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-03-25T18:21:30.510 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:30.542 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:21:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:30.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:21:30.576 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:21:30.576 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi046 2024-03-25T18:21:30.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:21:30.617 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253/remote/smithi164 2024-03-25T18:21:30.618 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:21:30.652 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:21:30.653 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:21:30.655 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:21:30.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:21:30.714 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:21:30.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:21:30.726 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:21:30.726 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:21:30.728 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:21:30.740 INFO:teuthology.orchestra.run.smithi046.stdout: 265625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:21 /home/ubuntu/cephtest 2024-03-25T18:21:30.756 INFO:teuthology.orchestra.run.smithi164.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:21 /home/ubuntu/cephtest 2024-03-25T18:21:30.758 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:21:30.769 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:21:30.798 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:21:30.826 DEBUG:teuthology.nuke:shortname: smithi046 2024-03-25T18:21:30.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:21:30.851 DEBUG:teuthology.nuke:shortname: smithi164 2024-03-25T18:21:30.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:21:30.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:05.077750', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGgMmjL9vIjb25dSI+q3qzntVgzJ5w3kec8X6xKB/SOOyIFMDrMxKtF8cO1JbTgGDF3xdWS63ZPWI7+5Td7VeE='} 2024-03-25T18:21:30.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621253', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:05.076152', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsZiptMAL9EmMGCc4pX0U3Pjy3haBJUASvF1lidQ3R5LIIt+aMAsppp9t5Nrftpp9rpOejdZcHctBsAY5nXXc4='} 2024-03-25T18:21:30.904 INFO:teuthology.orchestra.console.smithi046:Power off 2024-03-25T18:21:30.904 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:21:30.927 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-03-25T18:21:30.927 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:30.945 INFO:teuthology.orchestra.console.smithi164:Power off 2024-03-25T18:21:30.945 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:21:30.955 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-25T18:21:30.969 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-03-25T18:21:30.969 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:30.989 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-25T18:21:34.957 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:34.991 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:35.068 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-25T18:21:35.101 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-25T18:21:39.073 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:39.102 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:21:39.184 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-03-25T18:21:39.211 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-03-25T18:21:39.285 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-03-25T18:21:39.312 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-03-25T18:21:39.416 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7675.992205142975 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ba13d80b99034eb7b47f79a369718768 status: fail success: false 2024-03-25T18:21:39.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:21:39.493 INFO:teuthology.run:FAIL